HPO_Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Hyperparameter Optimization (HPO) of Machine Learning Models\n",
    "L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, pp. 295–316, 2020, doi: https://doi.org/10.1016/j.neucom.2020.07.061.\n",
    "\n",
    "### **Sample code for regression problems**  \n",
    "**Dataset used:**  \n",
    "&nbsp; Boson Housing dataset from sklearn\n",
    "\n",
    "**Machine learning algorithms used:**  \n",
    "&nbsp; Random forest (RF), support vector machine (SVM), k-nearest neighbor (KNN), artificial neural network (ANN)\n",
    "\n",
    "**HPO algorithms used:**  \n",
    "&nbsp; Grid search, random search, hyperband, Bayesian Optimization with Gaussian Processes (BO-GP), Bayesian Optimization with Tree-structured Parzen Estimator (BO-TPE), particle swarm optimization (PSO), genetic algorithm (GA).\n",
    "\n",
    "**Performance metric:**  \n",
    "&nbsp; Mean square error (MSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split,cross_val_score\n",
    "from sklearn.ensemble import RandomForestClassifier,RandomForestRegressor\n",
    "from sklearn.metrics import classification_report,confusion_matrix,accuracy_score\n",
    "from sklearn.neighbors import KNeighborsClassifier,KNeighborsRegressor\n",
    "from sklearn.svm import SVC,SVR\n",
    "from sklearn import datasets\n",
    "import scipy.stats as stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Boston Housing dataset\n",
    "We will take the Housing dataset which contains information about different houses in Boston. There are 506 samples and 13 feature variables in this Boston dataset. The main goal is to predict the value of prices of the house using the given features.\n",
    "\n",
    "You can read more about the data and the variables [[1]](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html) [[2]](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X, y = datasets.load_boston(return_X_y=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data': array([[6.3200e-03, 1.8000e+01, 2.3100e+00, ..., 1.5300e+01, 3.9690e+02,\n",
       "         4.9800e+00],\n",
       "        [2.7310e-02, 0.0000e+00, 7.0700e+00, ..., 1.7800e+01, 3.9690e+02,\n",
       "         9.1400e+00],\n",
       "        [2.7290e-02, 0.0000e+00, 7.0700e+00, ..., 1.7800e+01, 3.9283e+02,\n",
       "         4.0300e+00],\n",
       "        ...,\n",
       "        [6.0760e-02, 0.0000e+00, 1.1930e+01, ..., 2.1000e+01, 3.9690e+02,\n",
       "         5.6400e+00],\n",
       "        [1.0959e-01, 0.0000e+00, 1.1930e+01, ..., 2.1000e+01, 3.9345e+02,\n",
       "         6.4800e+00],\n",
       "        [4.7410e-02, 0.0000e+00, 1.1930e+01, ..., 2.1000e+01, 3.9690e+02,\n",
       "         7.8800e+00]]),\n",
       " 'target': array([24. , 21.6, 34.7, 33.4, 36.2, 28.7, 22.9, 27.1, 16.5, 18.9, 15. ,\n",
       "        18.9, 21.7, 20.4, 18.2, 19.9, 23.1, 17.5, 20.2, 18.2, 13.6, 19.6,\n",
       "        15.2, 14.5, 15.6, 13.9, 16.6, 14.8, 18.4, 21. , 12.7, 14.5, 13.2,\n",
       "        13.1, 13.5, 18.9, 20. , 21. , 24.7, 30.8, 34.9, 26.6, 25.3, 24.7,\n",
       "        21.2, 19.3, 20. , 16.6, 14.4, 19.4, 19.7, 20.5, 25. , 23.4, 18.9,\n",
       "        35.4, 24.7, 31.6, 23.3, 19.6, 18.7, 16. , 22.2, 25. , 33. , 23.5,\n",
       "        19.4, 22. , 17.4, 20.9, 24.2, 21.7, 22.8, 23.4, 24.1, 21.4, 20. ,\n",
       "        20.8, 21.2, 20.3, 28. , 23.9, 24.8, 22.9, 23.9, 26.6, 22.5, 22.2,\n",
       "        23.6, 28.7, 22.6, 22. , 22.9, 25. , 20.6, 28.4, 21.4, 38.7, 43.8,\n",
       "        33.2, 27.5, 26.5, 18.6, 19.3, 20.1, 19.5, 19.5, 20.4, 19.8, 19.4,\n",
       "        21.7, 22.8, 18.8, 18.7, 18.5, 18.3, 21.2, 19.2, 20.4, 19.3, 22. ,\n",
       "        20.3, 20.5, 17.3, 18.8, 21.4, 15.7, 16.2, 18. , 14.3, 19.2, 19.6,\n",
       "        23. , 18.4, 15.6, 18.1, 17.4, 17.1, 13.3, 17.8, 14. , 14.4, 13.4,\n",
       "        15.6, 11.8, 13.8, 15.6, 14.6, 17.8, 15.4, 21.5, 19.6, 15.3, 19.4,\n",
       "        17. , 15.6, 13.1, 41.3, 24.3, 23.3, 27. , 50. , 50. , 50. , 22.7,\n",
       "        25. , 50. , 23.8, 23.8, 22.3, 17.4, 19.1, 23.1, 23.6, 22.6, 29.4,\n",
       "        23.2, 24.6, 29.9, 37.2, 39.8, 36.2, 37.9, 32.5, 26.4, 29.6, 50. ,\n",
       "        32. , 29.8, 34.9, 37. , 30.5, 36.4, 31.1, 29.1, 50. , 33.3, 30.3,\n",
       "        34.6, 34.9, 32.9, 24.1, 42.3, 48.5, 50. , 22.6, 24.4, 22.5, 24.4,\n",
       "        20. , 21.7, 19.3, 22.4, 28.1, 23.7, 25. , 23.3, 28.7, 21.5, 23. ,\n",
       "        26.7, 21.7, 27.5, 30.1, 44.8, 50. , 37.6, 31.6, 46.7, 31.5, 24.3,\n",
       "        31.7, 41.7, 48.3, 29. , 24. , 25.1, 31.5, 23.7, 23.3, 22. , 20.1,\n",
       "        22.2, 23.7, 17.6, 18.5, 24.3, 20.5, 24.5, 26.2, 24.4, 24.8, 29.6,\n",
       "        42.8, 21.9, 20.9, 44. , 50. , 36. , 30.1, 33.8, 43.1, 48.8, 31. ,\n",
       "        36.5, 22.8, 30.7, 50. , 43.5, 20.7, 21.1, 25.2, 24.4, 35.2, 32.4,\n",
       "        32. , 33.2, 33.1, 29.1, 35.1, 45.4, 35.4, 46. , 50. , 32.2, 22. ,\n",
       "        20.1, 23.2, 22.3, 24.8, 28.5, 37.3, 27.9, 23.9, 21.7, 28.6, 27.1,\n",
       "        20.3, 22.5, 29. , 24.8, 22. , 26.4, 33.1, 36.1, 28.4, 33.4, 28.2,\n",
       "        22.8, 20.3, 16.1, 22.1, 19.4, 21.6, 23.8, 16.2, 17.8, 19.8, 23.1,\n",
       "        21. , 23.8, 23.1, 20.4, 18.5, 25. , 24.6, 23. , 22.2, 19.3, 22.6,\n",
       "        19.8, 17.1, 19.4, 22.2, 20.7, 21.1, 19.5, 18.5, 20.6, 19. , 18.7,\n",
       "        32.7, 16.5, 23.9, 31.2, 17.5, 17.2, 23.1, 24.5, 26.6, 22.9, 24.1,\n",
       "        18.6, 30.1, 18.2, 20.6, 17.8, 21.7, 22.7, 22.6, 25. , 19.9, 20.8,\n",
       "        16.8, 21.9, 27.5, 21.9, 23.1, 50. , 50. , 50. , 50. , 50. , 13.8,\n",
       "        13.8, 15. , 13.9, 13.3, 13.1, 10.2, 10.4, 10.9, 11.3, 12.3,  8.8,\n",
       "         7.2, 10.5,  7.4, 10.2, 11.5, 15.1, 23.2,  9.7, 13.8, 12.7, 13.1,\n",
       "        12.5,  8.5,  5. ,  6.3,  5.6,  7.2, 12.1,  8.3,  8.5,  5. , 11.9,\n",
       "        27.9, 17.2, 27.5, 15. , 17.2, 17.9, 16.3,  7. ,  7.2,  7.5, 10.4,\n",
       "         8.8,  8.4, 16.7, 14.2, 20.8, 13.4, 11.7,  8.3, 10.2, 10.9, 11. ,\n",
       "         9.5, 14.5, 14.1, 16.1, 14.3, 11.7, 13.4,  9.6,  8.7,  8.4, 12.8,\n",
       "        10.5, 17.1, 18.4, 15.4, 10.8, 11.8, 14.9, 12.6, 14.1, 13. , 13.4,\n",
       "        15.2, 16.1, 17.8, 14.9, 14.1, 12.7, 13.5, 14.9, 20. , 16.4, 17.7,\n",
       "        19.5, 20.2, 21.4, 19.9, 19. , 19.1, 19.1, 20.1, 19.9, 19.6, 23.2,\n",
       "        29.8, 13.8, 13.3, 16.7, 12. , 14.6, 21.4, 23. , 23.7, 25. , 21.8,\n",
       "        20.6, 21.2, 19.1, 20.6, 15.2,  7. ,  8.1, 13.6, 20.1, 21.8, 24.5,\n",
       "        23.1, 19.7, 18.3, 21.2, 17.5, 16.8, 22.4, 20.6, 23.9, 22. , 11.9]),\n",
       " 'feature_names': array(['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD',\n",
       "        'TAX', 'PTRATIO', 'B', 'LSTAT'], dtype='<U7'),\n",
       " 'DESCR': \".. _boston_dataset:\\n\\nBoston house prices dataset\\n---------------------------\\n\\n**Data Set Characteristics:**  \\n\\n    :Number of Instances: 506 \\n\\n    :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.\\n\\n    :Attribute Information (in order):\\n        - CRIM     per capita crime rate by town\\n        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.\\n        - INDUS    proportion of non-retail business acres per town\\n        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\\n        - NOX      nitric oxides concentration (parts per 10 million)\\n        - RM       average number of rooms per dwelling\\n        - AGE      proportion of owner-occupied units built prior to 1940\\n        - DIS      weighted distances to five Boston employment centres\\n        - RAD      index of accessibility to radial highways\\n        - TAX      full-value property-tax rate per $10,000\\n        - PTRATIO  pupil-teacher ratio by town\\n        - B        1000(Bk - 0.63)^2 where Bk is the proportion of black people by town\\n        - LSTAT    % lower status of the population\\n        - MEDV     Median value of owner-occupied homes in $1000's\\n\\n    :Missing Attribute Values: None\\n\\n    :Creator: Harrison, D. and Rubinfeld, D.L.\\n\\nThis is a copy of UCI ML housing dataset.\\nhttps://archive.ics.uci.edu/ml/machine-learning-databases/housing/\\n\\n\\nThis dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.\\n\\nThe Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic\\nprices and the demand for clean air', J. Environ. Economics & Management,\\nvol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics\\n...', Wiley, 1980.   N.B. Various transformations are used in the table on\\npages 244-261 of the latter.\\n\\nThe Boston house-price data has been used in many machine learning papers that address regression\\nproblems.   \\n     \\n.. topic:: References\\n\\n   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.\\n   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.\\n\",\n",
       " 'filename': 'D:\\\\py36\\\\lib\\\\site-packages\\\\sklearn\\\\datasets\\\\data\\\\boston_house_prices.csv'}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = datasets.load_boston()\n",
    "d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1.0</td>\n",
       "      <td>296.0</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.90</td>\n",
       "      <td>4.98</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>396.90</td>\n",
       "      <td>9.14</td>\n",
       "      <td>21.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>392.83</td>\n",
       "      <td>4.03</td>\n",
       "      <td>34.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>394.63</td>\n",
       "      <td>2.94</td>\n",
       "      <td>33.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>54.2</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.33</td>\n",
       "      <td>36.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>501</th>\n",
       "      <td>0.06263</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.93</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.573</td>\n",
       "      <td>6.593</td>\n",
       "      <td>69.1</td>\n",
       "      <td>2.4786</td>\n",
       "      <td>1.0</td>\n",
       "      <td>273.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>391.99</td>\n",
       "      <td>9.67</td>\n",
       "      <td>22.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>502</th>\n",
       "      <td>0.04527</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.93</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.573</td>\n",
       "      <td>6.120</td>\n",
       "      <td>76.7</td>\n",
       "      <td>2.2875</td>\n",
       "      <td>1.0</td>\n",
       "      <td>273.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>396.90</td>\n",
       "      <td>9.08</td>\n",
       "      <td>20.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>503</th>\n",
       "      <td>0.06076</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.93</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.573</td>\n",
       "      <td>6.976</td>\n",
       "      <td>91.0</td>\n",
       "      <td>2.1675</td>\n",
       "      <td>1.0</td>\n",
       "      <td>273.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.64</td>\n",
       "      <td>23.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>0.10959</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.93</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.573</td>\n",
       "      <td>6.794</td>\n",
       "      <td>89.3</td>\n",
       "      <td>2.3889</td>\n",
       "      <td>1.0</td>\n",
       "      <td>273.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>393.45</td>\n",
       "      <td>6.48</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>505</th>\n",
       "      <td>0.04741</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.93</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.573</td>\n",
       "      <td>6.030</td>\n",
       "      <td>80.8</td>\n",
       "      <td>2.5050</td>\n",
       "      <td>1.0</td>\n",
       "      <td>273.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>396.90</td>\n",
       "      <td>7.88</td>\n",
       "      <td>11.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>506 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD    TAX  \\\n",
       "0    0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900  1.0  296.0   \n",
       "1    0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671  2.0  242.0   \n",
       "2    0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671  2.0  242.0   \n",
       "3    0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622  3.0  222.0   \n",
       "4    0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622  3.0  222.0   \n",
       "..       ...   ...    ...   ...    ...    ...   ...     ...  ...    ...   \n",
       "501  0.06263   0.0  11.93   0.0  0.573  6.593  69.1  2.4786  1.0  273.0   \n",
       "502  0.04527   0.0  11.93   0.0  0.573  6.120  76.7  2.2875  1.0  273.0   \n",
       "503  0.06076   0.0  11.93   0.0  0.573  6.976  91.0  2.1675  1.0  273.0   \n",
       "504  0.10959   0.0  11.93   0.0  0.573  6.794  89.3  2.3889  1.0  273.0   \n",
       "505  0.04741   0.0  11.93   0.0  0.573  6.030  80.8  2.5050  1.0  273.0   \n",
       "\n",
       "     PTRATIO       B  LSTAT  Price  \n",
       "0       15.3  396.90   4.98   24.0  \n",
       "1       17.8  396.90   9.14   21.6  \n",
       "2       17.8  392.83   4.03   34.7  \n",
       "3       18.7  394.63   2.94   33.4  \n",
       "4       18.7  396.90   5.33   36.2  \n",
       "..       ...     ...    ...    ...  \n",
       "501     21.0  391.99   9.67   22.4  \n",
       "502     21.0  396.90   9.08   20.6  \n",
       "503     21.0  396.90   5.64   23.9  \n",
       "504     21.0  393.45   6.48   22.0  \n",
       "505     21.0  396.90   7.88   11.9  \n",
       "\n",
       "[506 rows x 14 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(data=d['data'], columns = d['feature_names'])\n",
    "df['Price'] = d['target']\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline Machine Learning Models: Regressors with Default Hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using 3-Fold Cross-Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE:32.42882531816474\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "clf = RandomForestRegressor()\n",
    "scores = cross_val_score(clf, X, y, cv=3,scoring='neg_mean_squared_error') # 3-fold cross-validation\n",
    "print(\"MSE:\"+ str(-scores.mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE:77.42951812579331\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "clf = SVR()\n",
    "scores = cross_val_score(clf, X, y, cv=3,scoring='neg_mean_squared_error')\n",
    "print(\"MSE:\"+ str(-scores.mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE:81.48773186343571\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "clf = KNeighborsRegressor()\n",
    "scores = cross_val_score(clf, X, y, cv=3,scoring='neg_mean_squared_error')\n",
    "print(\"MSE:\"+ str(-scores.mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Dense, Input\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from keras.wrappers.scikit_learn import KerasRegressor\n",
    "from keras.callbacks import EarlyStopping\n",
    "def ANN(optimizer = 'adam',neurons=32,batch_size=32,epochs=50,activation='relu',patience=5,loss='mse'):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(neurons, input_shape=(X.shape[1],), activation=activation))\n",
    "    model.add(Dense(neurons, activation=activation))\n",
    "    model.add(Dense(1))\n",
    "    model.compile(optimizer = optimizer, loss=loss)\n",
    "    early_stopping = EarlyStopping(monitor=\"loss\", patience = patience)# early stop patience\n",
    "    history = model.fit(X, y,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              callbacks = [early_stopping],\n",
    "              verbose=0) #verbose set to 1 will show the training process\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE:43.28833796634842\n"
     ]
    }
   ],
   "source": [
    "clf = KerasRegressor(build_fn=ANN, verbose=0)\n",
    "scores = cross_val_score(clf, X, y, cv=3,scoring='neg_mean_squared_error')\n",
    "print(\"MSE:\"+ str(-scores.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HPO Algorithm 1: Grid Search\n",
    "Search all the given hyper-parameter configurations\n",
    "\n",
    "**Advantages:**\n",
    "* Simple implementation.  \n",
    "\n",
    "**Disadvantages:**  \n",
    "* Time-consuming,\n",
    "* Only efficient with categorical HPs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': 20, 'max_depth': 15}\n",
      "MSE:29.03074736502433\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "# Define the hyperparameter configuration space\n",
    "rf_params = {\n",
    "    'n_estimators': [10, 20, 30],\n",
    "    #'max_features': ['sqrt',0.5],\n",
    "    'max_depth': [15,20,30,50],\n",
    "    #'min_samples_leaf': [1,2,4,8],\n",
    "    #\"bootstrap\":[True,False],\n",
    "    #\"criterion\":['mse','mae']\n",
    "}\n",
    "clf = RandomForestRegressor(random_state=0)\n",
    "grid = GridSearchCV(clf, rf_params, cv=3, scoring='neg_mean_squared_error')\n",
    "grid.fit(X, y)\n",
    "print(grid.best_params_)\n",
    "print(\"MSE:\"+ str(-grid.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'kernel': 'poly', 'epsilon': 0.01, 'C': 100}\n",
      "MSE:67.07483887754718\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "rf_params = {\n",
    "    'C': [1,10, 100],\n",
    "    \"kernel\":['poly','rbf','sigmoid'],\n",
    "    \"epsilon\":[0.01,0.1,1]\n",
    "}\n",
    "clf = SVR(gamma='scale')\n",
    "grid = GridSearchCV(clf, rf_params, cv=3, scoring='neg_mean_squared_error')\n",
    "grid.fit(X, y)\n",
    "print(grid.best_params_)\n",
    "print(\"MSE:\"+ str(-grid.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 5}\n",
      "MSE:81.52933517786562\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "rf_params = {\n",
    "    'n_neighbors': [2, 3, 5, 7, 10]\n",
    "}\n",
    "clf = KNeighborsRegressor()\n",
    "grid = GridSearchCV(clf, rf_params, cv=3, scoring='neg_mean_squared_error')\n",
    "grid.fit(X, y)\n",
    "print(grid.best_params_)\n",
    "print(\"MSE:\"+ str(-grid.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'patience': 5, 'optimizer': 'adam', 'batch_size': 16, 'loss': 'mse', 'activation': 'relu', 'neurons': 32, 'epochs': 50}\n",
      "MSE:52.18262109619083\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "rf_params = {\n",
    "    'optimizer': ['adam','rmsprop'],\n",
    "    'activation': ['relu','tanh'],\n",
    "    'loss': ['mse','mae'],\n",
    "    'batch_size': [16,32],\n",
    "    'neurons':[16,32],\n",
    "    'epochs':[20,50],\n",
    "    'patience':[2,5]\n",
    "}\n",
    "clf = KerasRegressor(build_fn=ANN, verbose=0)\n",
    "grid = GridSearchCV(clf, rf_params, cv=3,scoring='neg_mean_squared_error')\n",
    "grid.fit(X, y)\n",
    "print(grid.best_params_)\n",
    "print(\"MSE:\"+ str(-grid.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HPO Algorithm 2: Random Search\n",
    "Randomly search hyper-parameter combinations in the search space\n",
    "\n",
    "**Advantages:**\n",
    "* More efficient than GS.\n",
    "* Enable parallelization. \n",
    "\n",
    "**Disadvantages:**  \n",
    "* Not consider previous results.\n",
    "* Not efficient with conditional HPs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'criterion': 'mse', 'min_samples_leaf': 1, 'min_samples_split': 7, 'n_estimators': 98, 'max_depth': 14, 'max_features': 5}\n",
      "MSE:26.377428606830506\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from scipy.stats import randint as sp_randint\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "# Define the hyperparameter configuration space\n",
    "rf_params = {\n",
    "    'n_estimators': sp_randint(10,100),\n",
    "    \"max_features\":sp_randint(1,13),\n",
    "    'max_depth': sp_randint(5,50),\n",
    "    \"min_samples_split\":sp_randint(2,11),\n",
    "    \"min_samples_leaf\":sp_randint(1,11),\n",
    "    \"criterion\":['mse','mae']\n",
    "}\n",
    "n_iter_search=20 #number of iterations is set to 20, you can increase this number if time permits\n",
    "clf = RandomForestRegressor(random_state=0)\n",
    "Random = RandomizedSearchCV(clf, param_distributions=rf_params,n_iter=n_iter_search,cv=3,scoring='neg_mean_squared_error')\n",
    "Random.fit(X, y)\n",
    "print(Random.best_params_)\n",
    "print(\"MSE:\"+ str(-Random.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'kernel': 'poly', 'epsilon': 0.49678099309788193, 'C': 27.417074148575495}\n",
      "MSE:60.03157881614154\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from scipy.stats import randint as sp_randint\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "rf_params = {\n",
    "    'C': stats.uniform(0,50),\n",
    "    \"kernel\":['poly','rbf','sigmoid'],\n",
    "    \"epsilon\":stats.uniform(0,1)\n",
    "}\n",
    "n_iter_search=20\n",
    "clf = SVR(gamma='scale')\n",
    "Random = RandomizedSearchCV(clf, param_distributions=rf_params,n_iter=n_iter_search,cv=3,scoring='neg_mean_squared_error')\n",
    "Random.fit(X, y)\n",
    "print(Random.best_params_)\n",
    "print(\"MSE:\"+ str(-Random.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 13}\n",
      "MSE:80.7723025469514\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from scipy.stats import randint as sp_randint\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "rf_params = {\n",
    "    'n_neighbors': sp_randint(1,20),\n",
    "}\n",
    "n_iter_search=10\n",
    "clf = KNeighborsRegressor()\n",
    "Random = RandomizedSearchCV(clf, param_distributions=rf_params,n_iter=n_iter_search,cv=3,scoring='neg_mean_squared_error')\n",
    "Random.fit(X, y)\n",
    "print(Random.best_params_)\n",
    "print(\"MSE:\"+ str(-Random.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'activation': 'relu', 'optimizer': 'adam', 'batch_size': 64, 'neurons': 72, 'epochs': 50, 'patience': 9, 'loss': 'mse'}\n",
      "MSE:53.522195500716826\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "from scipy.stats import randint as sp_randint\n",
    "from random import randrange as sp_randrange\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "rf_params = {\n",
    "    'optimizer': ['adam','rmsprop'],\n",
    "    'activation': ['relu','tanh'],\n",
    "    'loss': ['mse','mae'],\n",
    "    'batch_size': [16,32,64],\n",
    "    'neurons':sp_randint(10,100),\n",
    "    'epochs':[20,50],\n",
    "    #'epochs':[20,50,100,200],\n",
    "    'patience':sp_randint(3,20)\n",
    "}\n",
    "n_iter_search=10\n",
    "clf = KerasRegressor(build_fn=ANN, verbose=0)\n",
    "Random = RandomizedSearchCV(clf, param_distributions=rf_params,n_iter=n_iter_search,cv=3,scoring='neg_mean_squared_error')\n",
    "Random.fit(X, y)\n",
    "print(Random.best_params_)\n",
    "print(\"MSE:\"+ str(-Random.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HPO Algorithm 3: Hyperband\n",
    "Generate small-sized subsets and allocate budgets to each hyper-parameter combination based on its performance\n",
    "\n",
    "**Advantages:**\n",
    "* Enable parallelization.  \n",
    "\n",
    "**Disadvantages:**  \n",
    "* Not efficient with conditional HPs.\n",
    "* Require subsets with small budgets to be representative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'criterion': 'mae', 'min_samples_leaf': 3, 'min_samples_split': 6, 'max_features': 6, 'n_estimators': 11, 'max_depth': 21}\n",
      "MSE:26.44144227942378\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from hyperband import HyperbandSearchCV\n",
    "from scipy.stats import randint as sp_randint\n",
    "# Define the hyperparameter configuration space\n",
    "rf_params = {\n",
    "    'n_estimators': sp_randint(10,100),\n",
    "    \"max_features\":sp_randint(1,13),\n",
    "    'max_depth': sp_randint(5,50),\n",
    "    \"min_samples_split\":sp_randint(2,11),\n",
    "    \"min_samples_leaf\":sp_randint(1,11),\n",
    "    \"criterion\":['mse','mae']\n",
    "}\n",
    "clf = RandomForestRegressor(random_state=0)\n",
    "hyper = HyperbandSearchCV(clf, param_distributions =rf_params,cv=3,min_iter=10,max_iter=100,scoring='neg_mean_squared_error')\n",
    "hyper.fit(X, y)\n",
    "print(hyper.best_params_)\n",
    "print(\"MSE:\"+ str(-hyper.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'kernel': 'poly', 'epsilon': 0.4490042156616516, 'C': 10}\n",
      "MSE:70.78132735518886\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from hyperband import HyperbandSearchCV\n",
    "from scipy.stats import randint as sp_randint\n",
    "rf_params = {\n",
    "    'C': stats.uniform(0,50),\n",
    "    \"kernel\":['poly','rbf','sigmoid'],\n",
    "    \"epsilon\":stats.uniform(0,1)\n",
    "}\n",
    "clf = SVR(gamma='scale')\n",
    "hyper = HyperbandSearchCV(clf, param_distributions =rf_params,cv=3,min_iter=1,max_iter=10,scoring='neg_mean_squared_error',resource_param='C')\n",
    "hyper.fit(X, y)\n",
    "print(hyper.best_params_)\n",
    "print(\"MSE:\"+ str(-hyper.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 6}\n",
      "MSE:80.87024044795783\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from hyperband import HyperbandSearchCV\n",
    "from scipy.stats import randint as sp_randint\n",
    "rf_params = {\n",
    "    'n_neighbors': range(1,20),\n",
    "}\n",
    "clf = KNeighborsRegressor()\n",
    "hyper = HyperbandSearchCV(clf, param_distributions =rf_params,cv=3,min_iter=1,max_iter=20,scoring='neg_mean_squared_error',resource_param='n_neighbors')\n",
    "hyper.fit(X, y)\n",
    "print(hyper.best_params_)\n",
    "print(\"MSE:\"+ str(-hyper.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'patience': 3, 'batch_size': 16, 'loss': 'mse', 'activation': 'relu', 'optimizer': 'adam', 'neurons': 48, 'epochs': 10}\n",
      "MSE:56.59321886927081\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "from hyperband import HyperbandSearchCV\n",
    "from scipy.stats import randint as sp_randint\n",
    "rf_params = {\n",
    "    'optimizer': ['adam','rmsprop'],\n",
    "    'activation': ['relu','tanh'],\n",
    "    'loss': ['mse','mae'],\n",
    "    'batch_size': [16,32,64],\n",
    "    'neurons':sp_randint(10,100),\n",
    "    'epochs':[20,50],\n",
    "    #'epochs':[20,50,100,200],\n",
    "    'patience':sp_randint(3,20)\n",
    "}\n",
    "clf = KerasRegressor(build_fn=ANN, epochs=20, verbose=0)\n",
    "hyper = HyperbandSearchCV(clf, param_distributions =rf_params,cv=3,min_iter=1,max_iter=10,scoring='neg_mean_squared_error',resource_param='epochs')\n",
    "hyper.fit(X, y)\n",
    "print(hyper.best_params_)\n",
    "print(\"MSE:\"+ str(-hyper.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HPO Algorithm 4: BO-GP\n",
    "Bayesian Optimization with Gaussian Process (BO-GP)\n",
    "\n",
    "**Advantages:**\n",
    "* Fast convergence speed for continuous HPs.  \n",
    "\n",
    "**Disadvantages:**  \n",
    "* Poor capacity for parallelization.\n",
    "* Not efficient with conditional HPs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using skopt.BayesSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'criterion': 'mse', 'min_samples_leaf': 1, 'min_samples_split': 11, 'max_depth': 38, 'n_estimators': 86, 'max_features': 8}\n",
      "MSE:26.138895388690205\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from skopt import Optimizer\n",
    "from skopt import BayesSearchCV \n",
    "from skopt.space import Real, Categorical, Integer\n",
    "# Define the hyperparameter configuration space\n",
    "rf_params = {\n",
    "    'n_estimators': Integer(10,100),\n",
    "    \"max_features\":Integer(1,13),\n",
    "    'max_depth': Integer(5,50),\n",
    "    \"min_samples_split\":Integer(2,11),\n",
    "    \"min_samples_leaf\":Integer(1,11),\n",
    "    \"criterion\":['mse','mae']\n",
    "}\n",
    "clf = RandomForestRegressor(random_state=0)\n",
    "Bayes = BayesSearchCV(clf, rf_params,cv=3,n_iter=20, scoring='neg_mean_squared_error') \n",
    "#number of iterations is set to 20, you can increase this number if time permits\n",
    "Bayes.fit(X, y)\n",
    "print(Bayes.best_params_)\n",
    "bclf = Bayes.best_estimator_\n",
    "print(\"MSE:\"+ str(-Bayes.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'kernel': 'poly', 'epsilon': 0.16781739810509447, 'C': 43.14510166511289}\n",
      "MSE:59.52440851660976\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from skopt import Optimizer\n",
    "from skopt import BayesSearchCV \n",
    "from skopt.space import Real, Categorical, Integer\n",
    "rf_params = {\n",
    "    'C': Real(0,50),\n",
    "    \"kernel\":['poly','rbf','sigmoid'],\n",
    "    'epsilon': Real(0,1)\n",
    "}\n",
    "clf = SVR(gamma='scale')\n",
    "Bayes = BayesSearchCV(clf, rf_params,cv=3,n_iter=20, scoring='neg_mean_squared_error')\n",
    "Bayes.fit(X, y)\n",
    "print(Bayes.best_params_)\n",
    "print(\"MSE:\"+ str(-Bayes.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 13}\n",
      "MSE:80.7723025469514\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from skopt import Optimizer\n",
    "from skopt import BayesSearchCV \n",
    "from skopt.space import Real, Categorical, Integer\n",
    "rf_params = {\n",
    "    'n_neighbors': Integer(1,20),\n",
    "}\n",
    "clf = KNeighborsRegressor()\n",
    "Bayes = BayesSearchCV(clf, rf_params,cv=3,n_iter=10, scoring='neg_mean_squared_error')\n",
    "Bayes.fit(X, y)\n",
    "print(Bayes.best_params_)\n",
    "print(\"MSE:\"+ str(-Bayes.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'patience': 15, 'optimizer': 'rmsprop', 'batch_size': 32, 'loss': 'mae', 'activation': 'relu', 'neurons': 43, 'epochs': 38}\n",
      "MSE:63.93545981123919\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "from skopt import Optimizer\n",
    "from skopt import BayesSearchCV \n",
    "from skopt.space import Real, Categorical, Integer\n",
    "rf_params = {\n",
    "    'optimizer': ['adam','rmsprop'],\n",
    "    'activation': ['relu','tanh'],\n",
    "    'loss': ['mse','mae'],\n",
    "    'batch_size': [16,32,64],\n",
    "    'neurons':Integer(10,100),\n",
    "    'epochs':[20,50],\n",
    "    #'epochs':[20,50,100,200],\n",
    "    'patience':Integer(3,20)\n",
    "}\n",
    "clf = KerasRegressor(build_fn=ANN, verbose=0)\n",
    "Bayes = BayesSearchCV(clf, rf_params,cv=3,n_iter=10, scoring='neg_mean_squared_error')\n",
    "Bayes.fit(X, y)\n",
    "print(Bayes.best_params_)\n",
    "print(\"MSE:\"+ str(-Bayes.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using skopt.gp_minimize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE:26.4279\n",
      "[100, 50, 8, 11, 1, 'mse']\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from skopt.space import Real, Integer\n",
    "from skopt.utils import use_named_args\n",
    "\n",
    "reg = RandomForestRegressor()\n",
    "# Define the hyperparameter configuration space\n",
    "space  = [Integer(10, 100, name='n_estimators'),\n",
    "            Integer(5, 50, name='max_depth'),\n",
    "          Integer(1, 13, name='max_features'),\n",
    "          Integer(2, 11, name='min_samples_split'),\n",
    "          Integer(1, 11, name='min_samples_leaf'),\n",
    "         Categorical(['mse', 'mae'], name='criterion')\n",
    "         ]\n",
    "# Define the objective function\n",
    "@use_named_args(space)\n",
    "def objective(**params):\n",
    "    reg.set_params(**params)\n",
    "\n",
    "    return -np.mean(cross_val_score(reg, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"neg_mean_squared_error\"))\n",
    "from skopt import gp_minimize\n",
    "res_gp = gp_minimize(objective, space, n_calls=20, random_state=0)\n",
    "#number of iterations is set to 20, you can increase this number if time permits\n",
    "print(\"MSE:%.4f\" % res_gp.fun)\n",
    "print(res_gp.x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE:61.2510\n",
      "[37.93078121611787, 'poly', 0.47360041934665753]\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from skopt.space import Real, Integer\n",
    "from skopt.utils import use_named_args\n",
    "\n",
    "reg = SVR(gamma='scale')\n",
    "space  = [Real(0, 50, name='C'),\n",
    "          Categorical(['poly','rbf','sigmoid'], name='kernel'),\n",
    "          Real(0, 1, name='epsilon'),\n",
    "         ]\n",
    "\n",
    "@use_named_args(space)\n",
    "def objective(**params):\n",
    "    reg.set_params(**params)\n",
    "\n",
    "    return -np.mean(cross_val_score(reg, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"neg_mean_squared_error\"))\n",
    "from skopt import gp_minimize\n",
    "res_gp = gp_minimize(objective, space, n_calls=20, random_state=0)\n",
    "print(\"MSE:%.4f\" % res_gp.fun)\n",
    "print(res_gp.x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE:80.7412\n",
      "[13]\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from skopt.space import Real, Integer\n",
    "from skopt.utils import use_named_args\n",
    "\n",
    "reg = KNeighborsRegressor()\n",
    "space  = [Integer(1, 20, name='n_neighbors')]\n",
    "\n",
    "@use_named_args(space)\n",
    "def objective(**params):\n",
    "    reg.set_params(**params)\n",
    "\n",
    "    return -np.mean(cross_val_score(reg, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"neg_mean_squared_error\"))\n",
    "from skopt import gp_minimize\n",
    "res_gp = gp_minimize(objective, space, n_calls=10, random_state=0)\n",
    "print(\"MSE:%.4f\" % res_gp.fun)\n",
    "print(res_gp.x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HPO Algorithm 5: BO-TPE\n",
    "Bayesian Optimization with Tree-structured Parzen Estimator (TPE)\n",
    "\n",
    "**Advantages:**\n",
    "* Efficient with all types of HPs.\n",
    "* Keep conditional dependencies.\n",
    "\n",
    "**Disadvantages:**  \n",
    "* Poor capacity for parallelization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████| 20/20 [00:05<00:00,  3.38it/s, best loss: 26.87220311513259]\n",
      "Random Forest: Hyperopt estimated optimum {'criterion': 0, 'min_samples_leaf': 3.0, 'min_samples_split': 5.0, 'max_depth': 50.0, 'n_estimators': 46.0, 'max_features': 8.0}\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "# Define the objective function\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_estimators': int(params['n_estimators']), \n",
    "        'max_depth': int(params['max_depth']),\n",
    "        'max_features': int(params['max_features']),\n",
    "        \"min_samples_split\":int(params['min_samples_split']),\n",
    "        \"min_samples_leaf\":int(params['min_samples_leaf']),\n",
    "        \"criterion\":str(params['criterion'])\n",
    "    }\n",
    "    clf = RandomForestRegressor( **params)\n",
    "    score = -np.mean(cross_val_score(clf, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"neg_mean_squared_error\"))\n",
    "\n",
    "    return {'loss':score, 'status': STATUS_OK }\n",
    "# Define the hyperparameter configuration space\n",
    "space = {\n",
    "    'n_estimators': hp.quniform('n_estimators', 10, 100, 1),\n",
    "    'max_depth': hp.quniform('max_depth', 5, 50, 1),\n",
    "    \"max_features\":hp.quniform('max_features', 1, 13, 1),\n",
    "    \"min_samples_split\":hp.quniform('min_samples_split',2,11,1),\n",
    "    \"min_samples_leaf\":hp.quniform('min_samples_leaf',1,11,1),\n",
    "    \"criterion\":hp.choice('criterion',['mse','mae'])\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=20)\n",
    "print(\"Random Forest: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████| 20/20 [00:03<00:00,  5.10it/s, best loss: 63.07694330646493]\n",
      "SVM: Hyperopt estimated optimum {'kernel': 0, 'epsilon': 0.6784908715313539, 'C': 50.9909224663369}\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'C': abs(float(params['C'])), \n",
    "        \"kernel\":str(params['kernel']),\n",
    "        'epsilon': abs(float(params['epsilon'])),\n",
    "    }\n",
    "    clf = SVR(gamma='scale', **params)\n",
    "    score = -np.mean(cross_val_score(clf, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"neg_mean_squared_error\"))\n",
    "    \n",
    "    return {'loss':score, 'status': STATUS_OK }\n",
    "\n",
    "space = {\n",
    "    'C': hp.normal('C', 0, 50),\n",
    "    \"kernel\":hp.choice('kernel',['poly','rbf','sigmoid']),\n",
    "    'epsilon': hp.normal('epsilon', 0, 1),\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=20)\n",
    "print(\"SVM: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████| 10/10 [00:00<00:00, 125.34it/s, best loss: 81.26511555604914]\n",
      "KNN: Hyperopt estimated optimum {'n_neighbors': 14.0}\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "def objective(params):\n",
    "    params = {\n",
    "        'n_neighbors': abs(int(params['n_neighbors']))\n",
    "    }\n",
    "    clf = KNeighborsRegressor( **params)\n",
    "    score = -np.mean(cross_val_score(clf, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"neg_mean_squared_error\"))\n",
    "\n",
    "    return {'loss':score, 'status': STATUS_OK }\n",
    "\n",
    "space = {\n",
    "    'n_neighbors': hp.quniform('n_neighbors', 1, 20, 1),\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=10)\n",
    "print(\"KNN: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████| 10/10 [04:50<00:00, 29.06s/it, best loss: 58.39425089889795]\n",
      "ANN: Hyperopt estimated optimum {'activation': 1, 'patience': 18.0, 'neurons': 80.0, 'batch_size': 32.0, 'loss': 0, 'epochs': 50.0, 'optimizer': 1}\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials\n",
    "from sklearn.model_selection import cross_val_score, StratifiedKFold\n",
    "def objective(params):\n",
    "    params = {\n",
    "        \"optimizer\":str(params['optimizer']),\n",
    "        \"activation\":str(params['activation']),\n",
    "        \"loss\":str(params['loss']),\n",
    "        'batch_size': abs(int(params['batch_size'])),\n",
    "        'neurons': abs(int(params['neurons'])),\n",
    "        'epochs': abs(int(params['epochs'])),\n",
    "        'patience': abs(int(params['patience']))\n",
    "    }\n",
    "    clf = KerasRegressor(build_fn=ANN,**params, verbose=0)\n",
    "    score = -np.mean(cross_val_score(clf, X, y, cv=3, \n",
    "                                    scoring=\"neg_mean_squared_error\"))\n",
    "\n",
    "    return {'loss':score, 'status': STATUS_OK }\n",
    "\n",
    "space = {\n",
    "    \"optimizer\":hp.choice('optimizer',['adam','rmsprop']),\n",
    "    \"activation\":hp.choice('activation',['relu','tanh']),\n",
    "    \"loss\":hp.choice('loss',['mse','mae']),\n",
    "    'batch_size': hp.quniform('batch_size', 16, 64, 16),\n",
    "    'neurons': hp.quniform('neurons', 10, 100, 10),\n",
    "    'epochs': hp.quniform('epochs', 20, 50, 10),\n",
    "    'patience': hp.quniform('patience', 3, 20, 3),\n",
    "}\n",
    "\n",
    "best = fmin(fn=objective,\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=10)\n",
    "print(\"ANN: Hyperopt estimated optimum {}\".format(best))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HPO Algorithm 6: PSO\n",
    "Partical swarm optimization (PSO): Each particle in a swarm communicates with other particles to detect and update the current global optimum in each iteration until the final optimum is detected.\n",
    "\n",
    "**Advantages:**\n",
    "* Efficient with all types of HPs.\n",
    "* Enable parallelization. \n",
    "\n",
    "**Disadvantages:**  \n",
    "* Require proper initialization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'max_depth': 37.745028056718674, 'min_samples_leaf': 1.2694935979785371, 'min_samples_split': 6.3674769629093, 'n_estimators': 97.73792799268917, 'max_features': 6.383593749999999}\n",
      "MSE:26.072547695737644\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "import optunity\n",
    "import optunity.metrics\n",
    "# Define the hyperparameter configuration space\n",
    "search = {\n",
    "    'n_estimators': [10, 100],\n",
    "    'max_features': [1, 13],\n",
    "    'max_depth': [5,50],\n",
    "    \"min_samples_split\":[2,11],\n",
    "    \"min_samples_leaf\":[1,11],\n",
    "         }\n",
    "# Define the objective function\n",
    "@optunity.cross_validated(x=X, y=y, num_folds=3)\n",
    "def performance(x_train, y_train, x_test, y_test,n_estimators=None, max_features=None,max_depth=None,min_samples_split=None,min_samples_leaf=None):\n",
    "    # fit the model\n",
    "    model = RandomForestRegressor(n_estimators=int(n_estimators),\n",
    "                                   max_features=int(max_features),\n",
    "                                   max_depth=int(max_depth),\n",
    "                                   min_samples_split=int(min_samples_split),\n",
    "                                   min_samples_leaf=int(min_samples_leaf),\n",
    "                                  )\n",
    "    scores=-np.mean(cross_val_score(model, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"neg_mean_squared_error\"))\n",
    "    return scores\n",
    "\n",
    "optimal_configuration, info, _ = optunity.minimize(performance,\n",
    "                                                  solver_name='particle swarm',\n",
    "                                                  num_evals=20,\n",
    "                                                   **search\n",
    "                                                  )\n",
    "print(optimal_configuration)\n",
    "print(\"MSE:\"+ str(info.optimum))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'kernel': 0.4658203125, 'epsilon': 0.4736328125, 'C': 25.341796875}\n",
      "MSE:60.25879498950017\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "import optunity\n",
    "import optunity.metrics\n",
    "search = {\n",
    "    'C': (0,50),\n",
    "    'kernel':[0,3],\n",
    "    'epsilon': (0, 1)\n",
    "         }\n",
    "@optunity.cross_validated(x=X, y=y, num_folds=3)\n",
    "def performance(x_train, y_train, x_test, y_test,C=None,kernel=None,epsilon=None):\n",
    "    # fit the model\n",
    "    if kernel<1:\n",
    "        ke='poly'\n",
    "    elif kernel<2:\n",
    "        ke='rbf'\n",
    "    else:\n",
    "        ke='sigmoid'\n",
    "    model = SVR(C=float(C),\n",
    "                kernel=ke,\n",
    "                gamma='scale',\n",
    "                epsilon=float(epsilon)\n",
    "                                  )\n",
    "\n",
    "    scores=-np.mean(cross_val_score(model, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"neg_mean_squared_error\"))\n",
    "\n",
    "    return scores\n",
    "\n",
    "optimal_configuration, info, _ = optunity.minimize(performance,\n",
    "                                                  solver_name='particle swarm',\n",
    "                                                  num_evals=20,\n",
    "                                                   **search\n",
    "                                                  )\n",
    "print(optimal_configuration)\n",
    "print(\"MSE:\"+ str(info.optimum))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 13.84912109375}\n",
      "MSE:80.74121499347262\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "import optunity\n",
    "import optunity.metrics\n",
    "search = {\n",
    "    'n_neighbors': [1, 20],\n",
    "         }\n",
    "@optunity.cross_validated(x=X, y=y, num_folds=3)\n",
    "def performance(x_train, y_train, x_test, y_test,n_neighbors=None):\n",
    "    # fit the model\n",
    "    model = KNeighborsRegressor(n_neighbors=int(n_neighbors),\n",
    "                                  )\n",
    "\n",
    "    scores=-np.mean(cross_val_score(model, X, y, cv=3, n_jobs=-1,\n",
    "                                    scoring=\"neg_mean_squared_error\"))\n",
    "\n",
    "    return scores\n",
    "\n",
    "optimal_configuration, info, _ = optunity.minimize(performance,\n",
    "                                                  solver_name='particle swarm',\n",
    "                                                  num_evals=10,\n",
    "                                                   **search\n",
    "                                                  )\n",
    "print(optimal_configuration)\n",
    "print(\"MSE:\"+ str(info.optimum))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'activation': 0.28555402710064154, 'neurons': 90.55370287990607, 'loss': 1.3789121219831286, 'batch_size': 0.6972214016007146, 'patience': 14.611429743835629, 'epochs': 35.81883196486749, 'optimizer': 1.1613324876783653}\n",
      "MSE:39.0195776769448\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "import optunity\n",
    "import optunity.metrics\n",
    "search = {\n",
    "    'optimizer':[0,2],\n",
    "    'activation':[0,2],\n",
    "    'loss':[0,2],\n",
    "    'batch_size': [0, 2],\n",
    "    'neurons': [10, 100],\n",
    "    'epochs': [20, 50],\n",
    "    'patience': [3, 20],\n",
    "         }\n",
    "@optunity.cross_validated(x=X, y=y, num_folds=3)\n",
    "def performance(x_train, y_train, x_test, y_test,optimizer=None,activation=None,loss=None,batch_size=None,neurons=None,epochs=None,patience=None):\n",
    "    # fit the model\n",
    "    if optimizer<1:\n",
    "        op='adam'\n",
    "    else:\n",
    "        op='rmsprop'\n",
    "    if activation<1:\n",
    "        ac='relu'\n",
    "    else:\n",
    "        ac='tanh'\n",
    "    if loss<1:\n",
    "        lo='mse'\n",
    "    else:\n",
    "        lo='mae'\n",
    "    if batch_size<1:\n",
    "        ba=16\n",
    "    else:\n",
    "        ba=32\n",
    "    model = ANN(optimizer=op,\n",
    "                activation=ac,\n",
    "                loss=lo,\n",
    "                batch_size=ba,\n",
    "                neurons=int(neurons),\n",
    "                epochs=int(epochs),\n",
    "                patience=int(patience)\n",
    "                                  )\n",
    "    clf = KerasRegressor(build_fn=ANN, verbose=0)\n",
    "    scores=-np.mean(cross_val_score(clf, X, y, cv=3, \n",
    "                                    scoring=\"neg_mean_squared_error\"))\n",
    "\n",
    "    return scores\n",
    "\n",
    "optimal_configuration, info, _ = optunity.minimize(performance,\n",
    "                                                  solver_name='particle swarm',\n",
    "                                                  num_evals=20,\n",
    "                                                   **search\n",
    "                                                  )\n",
    "print(optimal_configuration)\n",
    "print(\"MSE:\"+ str(info.optimum))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## HPO Algorithm 7: Genetic Algorithm\n",
    "Genetic algorithms detect well-performing hyper-parameter combinations in each generation, and pass them to the next generation until the best-performing combination is identified.\n",
    "\n",
    "**Advantages:**\n",
    "* Efficient with all types of HPs.\n",
    "* Not require good initialization.\n",
    " \n",
    "\n",
    "**Disadvantages:**  \n",
    "* Poor capacity for parallelization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using DEAP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Types [1, 1, 1, 1, 1, 1] and maxint [1, 89, 44, 8, 11, 9] detected\n",
      "--- Evolve in 8748000 possible combinations ---\n",
      "gen\tnevals\tavg     \tmin    \tmax     \tstd    \n",
      "0  \t10    \t-32.0014\t-54.374\t-27.5166\t7.55455\n",
      "1  \t6     \t-28.6069\t-31.6563\t-26.2084\t1.44316\n",
      "2  \t4     \t-27.7354\t-29.2441\t-26.2084\t1.11775\n",
      "3  \t1     \t-26.4502\t-27.5166\t-26.2084\t0.392132\n",
      "4  \t2     \t-26.2462\t-26.5859\t-26.2084\t0.113246\n",
      "5  \t9     \t-26.4494\t-27.7558\t-26.0528\t0.488179\n",
      "Best individual is: {'criterion': 'mse', 'n_estimators': 97, 'max_depth': 29, 'min_samples_split': 7, 'max_features': 8, 'min_samples_leaf': 3}\n",
      "with fitness: -26.052847277193734\n",
      "{'criterion': 'mse', 'n_estimators': 97, 'max_depth': 29, 'min_samples_split': 7, 'max_features': 8, 'min_samples_leaf': 3}\n",
      "MSE:26.052847277193734\n"
     ]
    }
   ],
   "source": [
    "#Random Forest\n",
    "from evolutionary_search import EvolutionaryAlgorithmSearchCV\n",
    "from scipy.stats import randint as sp_randint\n",
    "# Define the hyperparameter configuration space\n",
    "rf_params = {\n",
    "    'n_estimators': range(10,100),\n",
    "    \"max_features\":range(1,13),\n",
    "    'max_depth': range(5,50),\n",
    "    \"min_samples_split\":range(2,11),\n",
    "    \"min_samples_leaf\":range(1,11),\n",
    "    \"criterion\":['mse','mae']\n",
    "}\n",
    "clf = RandomForestRegressor(random_state=0)\n",
    "# Set the hyperparameters of GA    \n",
    "ga1 = EvolutionaryAlgorithmSearchCV(estimator=clf,\n",
    "                                   params=rf_params,\n",
    "                                   scoring=\"neg_mean_squared_error\",\n",
    "                                   cv=3,\n",
    "                                   verbose=1,\n",
    "                                   population_size=10,\n",
    "                                   gene_mutation_prob=0.10,\n",
    "                                   gene_crossover_prob=0.5,\n",
    "                                   tournament_size=3,\n",
    "                                   generations_number=5,\n",
    "                                   n_jobs=1)\n",
    "ga1.fit(X, y)\n",
    "print(ga1.best_params_)\n",
    "print(\"MSE:\"+ str(-ga1.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Types [2, 2, 1] and maxint [999, 99, 2] detected\n",
      "--- Evolve in 300000 possible combinations ---\n",
      "gen\tnevals\tavg     \tmin     \tmax     \tstd    \n",
      "0  \t10    \t-1622.54\t-7880.66\t-61.3248\t2655.55\n",
      "1  \t8     \t-68.6978\t-81.931 \t-61.0316\t8.46914\n",
      "2  \t5     \t-64.2231\t-78.2337\t-60.2699\t5.09324\n",
      "3  \t2     \t-61.5322\t-65.247 \t-60.2699\t1.32565\n",
      "4  \t10    \t-60.9099\t-62.4522\t-59.4154\t0.848694\n",
      "5  \t6     \t-60.1359\t-61.5717\t-59.4154\t0.689567\n",
      "Best individual is: {'C': 36.961021855583446, 'epsilon': 0.1079957723914794, 'kernel': 'poly'}\n",
      "with fitness: -59.41536985706522\n",
      "{'C': 36.961021855583446, 'epsilon': 0.1079957723914794, 'kernel': 'poly'}\n",
      "MSE:59.41536985706522\n"
     ]
    }
   ],
   "source": [
    "#SVM\n",
    "from evolutionary_search import EvolutionaryAlgorithmSearchCV\n",
    "rf_params = {\n",
    "    'C': np.random.uniform(0,50,1000),\n",
    "    \"kernel\":['poly','rbf','sigmoid'],\n",
    "    'epsilon': np.random.uniform(0,1,100),\n",
    "}\n",
    "clf = SVR(gamma='scale')\n",
    "ga1 = EvolutionaryAlgorithmSearchCV(estimator=clf,\n",
    "                                   params=rf_params,\n",
    "                                   scoring=\"neg_mean_squared_error\",\n",
    "                                   cv=3,\n",
    "                                   verbose=1,\n",
    "                                   population_size=10,\n",
    "                                   gene_mutation_prob=0.10,\n",
    "                                   gene_crossover_prob=0.5,\n",
    "                                   tournament_size=3,\n",
    "                                   generations_number=5,\n",
    "                                   n_jobs=1)\n",
    "ga1.fit(X, y)\n",
    "print(ga1.best_params_)\n",
    "print(\"MSE:\"+ str(-ga1.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Types [1] and maxint [18] detected\n",
      "--- Evolve in 19 possible combinations ---\n",
      "gen\tnevals\tavg     \tmin     \tmax     \tstd     \n",
      "0  \t10    \t-82.2005\t-82.7914\t-80.8702\t0.621814\n",
      "1  \t7     \t-81.5525\t-82.7021\t-80.8702\t0.599417\n",
      "2  \t8     \t-80.8702\t-80.8702\t-80.8702\t0       \n",
      "3  \t8     \t-80.8702\t-80.8702\t-80.8702\t0       \n",
      "4  \t10    \t-80.9662\t-81.8295\t-80.8702\t0.287764\n",
      "5  \t8     \t-80.8604\t-80.8702\t-80.7723\t0.0293814\n",
      "Best individual is: {'n_neighbors': 13}\n",
      "with fitness: -80.7723025469514\n",
      "{'n_neighbors': 13}\n",
      "MSE:80.7723025469514\n"
     ]
    }
   ],
   "source": [
    "#KNN\n",
    "from evolutionary_search import EvolutionaryAlgorithmSearchCV\n",
    "rf_params = {\n",
    "    'n_neighbors': range(1,20),\n",
    "}\n",
    "clf = KNeighborsRegressor()\n",
    "ga1 = EvolutionaryAlgorithmSearchCV(estimator=clf,\n",
    "                                   params=rf_params,\n",
    "                                   scoring=\"neg_mean_squared_error\",\n",
    "                                   cv=3,\n",
    "                                   verbose=1,\n",
    "                                   population_size=10,\n",
    "                                   gene_mutation_prob=0.10,\n",
    "                                   gene_crossover_prob=0.5,\n",
    "                                   tournament_size=3,\n",
    "                                   generations_number=5,\n",
    "                                   n_jobs=1)\n",
    "ga1.fit(X, y)\n",
    "print(ga1.best_params_)\n",
    "print(\"MSE:\"+ str(-ga1.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Types [1, 1, 1, 1, 1, 1, 1] and maxint [1, 16, 2, 1, 1, 89, 1] detected\n",
      "--- Evolve in 73440 possible combinations ---\n",
      "gen\tnevals\tavg     \tmin     \tmax    \tstd    \n",
      "0  \t10    \t-88.4899\t-157.552\t-50.727\t37.5649\n",
      "1  \t2     \t-60.9958\t-99.4323\t-50.727\t13.8181\n",
      "2  \t8     \t-74.4071\t-174.246\t-50.727\t42.0684\n",
      "3  \t7     \t-59.5146\t-138.603\t-50.727\t26.3628\n",
      "4  \t3     \t-54.6543\t-90.0003\t-50.727\t11.782 \n",
      "5  \t3     \t-50.727 \t-50.727 \t-50.727\t7.10543e-15\n",
      "Best individual is: {'patience': 5, 'batch_size': 32, 'loss': 'mse', 'activation': 'relu', 'optimizer': 'rmsprop', 'neurons': 30, 'epochs': 50}\n",
      "with fitness: -50.72700946933739\n",
      "{'patience': 5, 'batch_size': 32, 'loss': 'mse', 'activation': 'relu', 'optimizer': 'rmsprop', 'neurons': 30, 'epochs': 50}\n",
      "MSE:50.72700946933739\n"
     ]
    }
   ],
   "source": [
    "#ANN\n",
    "from evolutionary_search import EvolutionaryAlgorithmSearchCV\n",
    "# Define the hyperparameter configuration space\n",
    "rf_params = {\n",
    "    'optimizer': ['adam','rmsprop'],\n",
    "    'activation': ['relu','tanh'],\n",
    "    'loss': ['mse','mae'],\n",
    "    'batch_size': [16,32,64],\n",
    "    'neurons':range(10,100),\n",
    "    'epochs':[20,50],\n",
    "    #'epochs':[20,50,100,200],\n",
    "    'patience':range(3,20)\n",
    "}\n",
    "clf = KerasRegressor(build_fn=ANN, verbose=0)\n",
    "# Set the hyperparameters of GA    \n",
    "ga1 = EvolutionaryAlgorithmSearchCV(estimator=clf,\n",
    "                                   params=rf_params,\n",
    "                                   scoring=\"neg_mean_squared_error\",\n",
    "                                   cv=3,\n",
    "                                   verbose=1,\n",
    "                                   population_size=10,\n",
    "                                   gene_mutation_prob=0.10,\n",
    "                                   gene_crossover_prob=0.5,\n",
    "                                   tournament_size=3,\n",
    "                                   generations_number=5,\n",
    "                                   n_jobs=1)\n",
    "ga1.fit(X, y)\n",
    "print(ga1.best_params_)\n",
    "print(\"MSE:\"+ str(-ga1.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using TPOT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 operators have been imported by TPOT.\n",
      "Generation 1 - Current Pareto front scores:\n",
      "-1\t-27.250793005044518\tRandomForestRegressor(input_matrix, RandomForestRegressor__max_depth=78, RandomForestRegressor__max_features=5, RandomForestRegressor__min_samples_leaf=2, RandomForestRegressor__min_samples_split=9, RandomForestRegressor__n_estimators=168)\n",
      "-2\t-25.332016721242965\tRandomForestRegressor(RandomForestRegressor(input_matrix, RandomForestRegressor__max_depth=62, RandomForestRegressor__max_features=4, RandomForestRegressor__min_samples_leaf=5, RandomForestRegressor__min_samples_split=6, RandomForestRegressor__n_estimators=34), RandomForestRegressor__max_depth=78, RandomForestRegressor__max_features=5, RandomForestRegressor__min_samples_leaf=2, RandomForestRegressor__min_samples_split=9, RandomForestRegressor__n_estimators=168)\n",
      "\n",
      "Generation 2 - Current Pareto front scores:\n",
      "-1\t-27.250793005044518\tRandomForestRegressor(input_matrix, RandomForestRegressor__max_depth=78, RandomForestRegressor__max_features=5, RandomForestRegressor__min_samples_leaf=2, RandomForestRegressor__min_samples_split=9, RandomForestRegressor__n_estimators=168)\n",
      "-2\t-25.332016721242965\tRandomForestRegressor(RandomForestRegressor(input_matrix, RandomForestRegressor__max_depth=62, RandomForestRegressor__max_features=4, RandomForestRegressor__min_samples_leaf=5, RandomForestRegressor__min_samples_split=6, RandomForestRegressor__n_estimators=34), RandomForestRegressor__max_depth=78, RandomForestRegressor__max_features=5, RandomForestRegressor__min_samples_leaf=2, RandomForestRegressor__min_samples_split=9, RandomForestRegressor__n_estimators=168)\n",
      "\n",
      "Generation 3 - Current Pareto front scores:\n",
      "-1\t-27.250793005044518\tRandomForestRegressor(input_matrix, RandomForestRegressor__max_depth=78, RandomForestRegressor__max_features=5, RandomForestRegressor__min_samples_leaf=2, RandomForestRegressor__min_samples_split=9, RandomForestRegressor__n_estimators=168)\n",
      "-2\t-24.584400304272155\tRandomForestRegressor(RandomForestRegressor(input_matrix, RandomForestRegressor__max_depth=38, RandomForestRegressor__max_features=5, RandomForestRegressor__min_samples_leaf=7, RandomForestRegressor__min_samples_split=2, RandomForestRegressor__n_estimators=45), RandomForestRegressor__max_depth=81, RandomForestRegressor__max_features=5, RandomForestRegressor__min_samples_leaf=1, RandomForestRegressor__min_samples_split=7, RandomForestRegressor__n_estimators=103)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TPOTRegressor(config_dict={'sklearn.ensemble.RandomForestRegressor': {'max_depth': range(10, 100), 'min_samples_leaf': range(1, 11), 'min_samples_split': range(2, 11), 'max_features': range(1, 13), 'n_estimators': range(20, 200)}},\n",
       "       crossover_rate=0.1, cv=3, disable_update_check=False, early_stop=5,\n",
       "       generations=3, max_eval_time_mins=5, max_time_mins=None,\n",
       "       memory=None, mutation_rate=0.9, n_jobs=1, offspring_size=5,\n",
       "       periodic_checkpoint_folder=None, population_size=10,\n",
       "       random_state=None, scoring='neg_mean_squared_error', subsample=1.0,\n",
       "       template=None, use_dask=False, verbosity=3, warm_start=False)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Random Forest\n",
    "from tpot import TPOTRegressor\n",
    "# Define the hyperparameter configuration space\n",
    "parameters = {\n",
    "    'n_estimators': range(20,200),\n",
    "    \"max_features\":range(1,13),\n",
    "    'max_depth': range(10,100),\n",
    "    \"min_samples_split\":range(2,11),\n",
    "    \"min_samples_leaf\":range(1,11),\n",
    "    #\"criterion\":['mse','mae']\n",
    "             }\n",
    "# Set the hyperparameters of GA               \n",
    "ga2 = TPOTRegressor(generations= 3, population_size= 10, offspring_size= 5,\n",
    "                                 verbosity= 3, early_stop= 5,\n",
    "                                 config_dict=\n",
    "                                 {'sklearn.ensemble.RandomForestRegressor': parameters}, \n",
    "                                 cv = 3, scoring = 'neg_mean_squared_error')\n",
    "ga2.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 operators have been imported by TPOT.\n",
      "Generation 1 - Current Pareto front scores:\n",
      "-1\t-76.22342756605103\tSVR(CombineDFs(input_matrix, input_matrix), SVR__C=9.66289149097888, SVR__epsilon=0.6491076078954555, SVR__gamma=scale, SVR__kernel=rbf)\n",
      "-2\t-61.22653072468754\tSVR(SVR(CombineDFs(input_matrix, input_matrix), SVR__C=2.988848947755768, SVR__epsilon=0.31470079088508107, SVR__gamma=scale, SVR__kernel=poly), SVR__C=49.38466344757709, SVR__epsilon=0.9554593389709706, SVR__gamma=scale, SVR__kernel=poly)\n",
      "\n",
      "Generation 2 - Current Pareto front scores:\n",
      "-1\t-73.46312983258913\tSVR(input_matrix, SVR__C=9.66289149097888, SVR__epsilon=0.6491076078954555, SVR__gamma=scale, SVR__kernel=poly)\n",
      "-2\t-61.22653072468754\tSVR(SVR(CombineDFs(input_matrix, input_matrix), SVR__C=2.988848947755768, SVR__epsilon=0.31470079088508107, SVR__gamma=scale, SVR__kernel=poly), SVR__C=49.38466344757709, SVR__epsilon=0.9554593389709706, SVR__gamma=scale, SVR__kernel=poly)\n",
      "\n",
      "Generation 3 - Current Pareto front scores:\n",
      "-1\t-59.46539136594907\tSVR(input_matrix, SVR__C=40.42400106501475, SVR__epsilon=0.09227083238549583, SVR__gamma=scale, SVR__kernel=poly)\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TPOTRegressor(config_dict={'sklearn.svm.SVR': {'kernel': ['poly', 'rbf', 'sigmoid'], 'gamma': ['scale'], 'epsilon': array([0.25137, 0.0217 , ..., 0.7522 , 0.4685 ]), 'C': array([40.05907, 22.18248, ..., 33.62995,  2.73409])}},\n",
       "       crossover_rate=0.1, cv=3, disable_update_check=False, early_stop=5,\n",
       "       generations=3, max_eval_time_mins=5, max_time_mins=None,\n",
       "       memory=None, mutation_rate=0.9, n_jobs=1, offspring_size=5,\n",
       "       periodic_checkpoint_folder=None, population_size=10,\n",
       "       random_state=None, scoring='neg_mean_squared_error', subsample=1.0,\n",
       "       template=None, use_dask=False, verbosity=3, warm_start=False)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#SVM\n",
    "from tpot import TPOTRegressor\n",
    "\n",
    "parameters = {\n",
    "    'C': np.random.uniform(0,50,1000),\n",
    "    \"kernel\":['poly','rbf','sigmoid'],\n",
    "    'epsilon': np.random.uniform(0,1,100),\n",
    "    'gamma': ['scale']\n",
    "             }\n",
    "               \n",
    "ga2 = TPOTRegressor(generations= 3, population_size= 10, offspring_size= 5,\n",
    "                                 verbosity= 3, early_stop= 5,\n",
    "                                 config_dict=\n",
    "                                 {'sklearn.svm.SVR': parameters}, \n",
    "                                 cv = 3, scoring = 'neg_mean_squared_error')\n",
    "ga2.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 operators have been imported by TPOT.\n",
      "Generation 1 - Current Pareto front scores:\n",
      "-1\t-80.83005201647829\tKNeighborsRegressor(input_matrix, KNeighborsRegressor__n_neighbors=6)\n",
      "\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Generation 2 - Current Pareto front scores:\n",
      "-1\t-80.83005201647829\tKNeighborsRegressor(input_matrix, KNeighborsRegressor__n_neighbors=6)\n",
      "\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Generation 3 - Current Pareto front scores:\n",
      "-1\t-80.83005201647829\tKNeighborsRegressor(input_matrix, KNeighborsRegressor__n_neighbors=6)\n",
      "\n",
      "\n",
      "The optimized pipeline was not improved after evaluating 5 more generations. Will end the optimization process.\n",
      "\n",
      "TPOT closed prematurely. Will use the current best pipeline.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TPOTRegressor(config_dict={'sklearn.neighbors.KNeighborsRegressor': {'n_neighbors': range(1, 20)}},\n",
       "       crossover_rate=0.1, cv=3, disable_update_check=False, early_stop=5,\n",
       "       generations=3, max_eval_time_mins=5, max_time_mins=None,\n",
       "       memory=None, mutation_rate=0.9, n_jobs=1, offspring_size=5,\n",
       "       periodic_checkpoint_folder=None, population_size=10,\n",
       "       random_state=None, scoring='neg_mean_squared_error', subsample=1.0,\n",
       "       template=None, use_dask=False, verbosity=3, warm_start=False)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#KNN\n",
    "from tpot import TPOTRegressor\n",
    "\n",
    "parameters = {\n",
    "    'n_neighbors': range(1,20),\n",
    "             }\n",
    "               \n",
    "ga2 = TPOTRegressor(generations= 3, population_size= 10, offspring_size= 5,\n",
    "                                 verbosity= 3, early_stop= 5,\n",
    "                                 config_dict=\n",
    "                                 {'sklearn.neighbors.KNeighborsRegressor': parameters}, \n",
    "                                 cv = 3, scoring = 'neg_mean_squared_error')\n",
    "ga2.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "tf36",
   "language": "python",
   "name": "tf36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}