Autograd_and_optimizers.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/smlra-kjsce/Pytorch-101-2023/blob/main/Autograd_and_optimizers.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LqtBuCpHp0gY"
      },
      "source": [
        "# Forward & Backward Passes \n",
        "\n",
        "The forward pass of any neural network or for that matter a linear layer is just computing the predicted values of y from the given x value and the current weights.\n",
        "\n",
        "After the forward pass, loss is calculated. The backward pass deals with computing the gradients, i.e. the derivatives of the loss wrt the weights from the loss so that the weights can be updated for further iterations. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "duviVH7gqLLw"
      },
      "source": [
        "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAYkAAACACAYAAAD+pZXBAAAdNElEQVR4Ae2dvcsdRRvG8y8Egv9CSGP1FOlSBaxiGdI8YJnC0lIsAoKENKZ7G4NgZSEIFlYi2ASbVMEupY2FCIKQYl9+5+R6cmey3zu7O7t7DezZPfM918zc18zcM7vXKhsjYASMgBEwAg0IXGuwt7URMAJGwAgYgcok4UZgBIyAETACjQiYJBqhsYMRMAJGwAiYJNwGjIARMAJGoBEBk0SA5s8//6yePHlydT1+/Lh6/fp18LHeI/n69ddf18uAUzYCAYFXr16d+gl9ZGrbfP78efXo0aMQ+/yPX3zxRfXy5cv5E9pBCiaJUIk01ps3b56E8c8//1xxlUISdKIff/wx5NaPRmA9BBiw3L59u6LP8HxxcVF9++23ozJE+E8//XRU2LGBLi8vqxcvXowNfqhwJolQ3TT4+/fvB5vz43///XdqxHQKRiAyNG4ENw2cRvf9999fjU4gGEZYGMIrHPb37t2r7ty5UzEKkyFt4sIf7hjiU5qQBGExhCNt/HExA8JAaIQnbuxjJyDuu3fvntyUL8KRb/xT7r///vsUj3+MQBcCtL+HDx9eeeM/bQlDO8JN7UrtM3WTf9p+JAmeNVNRWAiI/oDBTm0YO7XrZ8+endz5SfsmdrjTnwhL/tQ/6DP4J57YDygHeVE5yJMM+cGeS+SIO/0OO5VN/rd8N0mE2qPRMJOg4dGgJJSpdDVQNTCC0TiuXbt2arQQAX4k+GksuGFogCIJprj4xWiqzjNCHP80NNzpOIzOMHQK3JQfNWbciE/pEE5TaBo49tgR/saNG6e4FIY78aij4MckcQWRHzoQoE3TPmnzEphqe7QlCVTuap9EyTNhMeoLtHX1D9qp3Olr6nekpf5AeiIJtV/iQ6DTjzD0X9IiL/QB7DUAVJ9RWNIUSZH29evXT3HQH1QO4lE5CI9MkJEfpYc9dqWsQiifY+8miYAcjebWrVunBkXjpMHQwNQ45FX/Y2PFjYakhgxJMPqnsWikgh/iowFjx2hGIw4acVyXpdGqsxCO/yIJGqg6JG6kpf+UAfIhLggPe5WB/JJHGeLkUmeRve9GoAsB2iajcvoJF21QAp2wuNMGuegvtEHIQII6xo894YlPAhd32i79g7D0FS6eiUPtHUGu/oRfkQ12XDIMiBQGO9JTu4ck4gCJ/+on5C2WA3suykSflT/iJA3Sj+ko/S3fTRKh9tRYg9Wp8YgUZK//aUPEHTc6CCMd7ghmhDWNG0MDpHHRGRD66jR0MI2O8EeHID8yND7CYSCJ2JnoHDR4/ItAaLyxI6gz0RGVJnGRB4iCfMc4la7vRqAOAdq2Bji4077VL2jzuCEsaYeMzGl/hKGtpYZ2i4Cln0Shiz/ipN2LjGivSkfuxEvbxZ/ip2+SDxkGbzFu+lcTSah/EZ8IibCRxPhPf2VQqTRJizCUnTxG4lE+tng3SYRaqyMJnKlwNTAaPo0ZU0cSCHOIAH90HPzSuDCxIym8BHZKEsTNCEaGODWToEPhLkP+iDvND/bqCPLLXQ04Todp8JGkon8/G4EUAQQzglaGvkO7wiA0+Y9Rm0dgcsnPyfHND37pN/IbR+IIXEiG/sdFeJET/9UXiYo4JLDpC5Ek6Etpn1HfiH2LPqE8EhflxChvpBn7DW7yf/L45oew6q/RfovPJolQa00koQ5Ap6BBaMSdCmWiolHFRsMIJgpfGjKkQUMnPjX4lCSIixENJMKMgEuNjmcaIWFp4OoM6kTEix/C0xGwp6Nhjx15wChe4mFEtJeRz6lw/pkVAbVz2g2CmktCFyFPH4htVG2LkTZutEXCIHzpX/jFiEgUF/0i9ieesZOhfzBo4k4ciiftm6RDXmnrtHvCKA3syQ8X8aufqRzYE79mI+SXvJNW7N/Y4Y806GekuQdjkkhqMR0lRGc19GhX9xzjiM/ySzxqQNE9Pke/etadBojgJw7FIzfuTfmM6co/dk3+5cd3I1CHAO2V9lfXbrFTu0rd5Rbt4zNpxXYd3eKz8hTbtdx1lx/dlSf9585Ai/RiPHInHoWJcco+5pMw+JV/xbH1u0ligzXIyEezmQ1m31k2AkUhwOxhb4I9J8AmiZxoLhQXDTqOahZK1skYgV0iwKzcphkBk0QzNnYxAkbACBweAZPE4ZuAATACRsAINCNgkmjGxi5GwAgYgcMjYJI4fBM4HgDodOJe/OMh4BIbgf4ImCT6Y2WfO0GA/fHsb7cxAkagGwGTRDdG9rEzBCAJzprYGAEj0I2ASaIbI/vYGQImiZ1VqIszKwImiVnhdeQlImCSKLFWnKdSETBJlFozztdsCJgkZoPWEe8QAZPEDivVRWpHwCTRjo9djUBEwCQR0fDzIRAwSRyiml3ITAiYJDIB6Wi2g8Aokvjnn6riwuj5r7/mKTTvEnr9+hw3aXD9+297Wn7/UDs+dh2NgEliNHQOuFUERpHEb79V1f/+dy4yApkttHMJZj7mI0L65puq+uqrbpL4+OOtVofzXTgCi5NEfP96fC4cJ2dvRwgMJok//qgqhPWrV2cU+P/0aX5EIAbS+Oyzt3F/+WU7GUFUXApDHOSPmYdmI29j85MRGIzAoiTBF9T4ohVfRuOu78EOzrUDGIEJCAwiic8/P6cUR+oQxosXE3JQExRBD/FwhxhkYrqy0508/PDDOS/ffXe2ZcaDXfh6m7z7bgTGILAYSfDJQc0c+MQfnwjk/Tn63OCYzDuMERiDQG+SYFSvJSaRBQl+8snbZH/55e3zlCeIgdE/gl9x8l8zBOL+6ad3U9A3piEFZg8Y9BfE4VnEu1j532gEFiOJ+GEPvgQlwhidcwc0AiMR6E0SCGlG9whtSAJBjAD+8MOqko6CWUUOQxrECVkQJ2mjiyBd7CEmZhnRMHsgP5AFBAJBEJY4NLOI/v1sBEYgsBhJKG+QBd+UtTECayHQmySmZBChjcBOL2YmUkpPid9hjcBCCCxGEno1M8tOjx49uioeegkbI7AkAr1JgtE9Ql0XAl/PujPTYDQ/1Xz99btxN6WlZSSWwkhb+ajzzzJUatLZSOru/0YgQWARkuD9/VpiQgfx+PHjUzaePXv23reaX716ddJT+MPkSU35bzYEepNEmmKqE0jd43/8SoDHO4K9aybBUtOQtIhPeoyYh7pnlq6uXTvrOkjDpFGHku0CAouQBOk9f/78pKzmmU4aFdkhP1ePT548qbhsjEBuBEaThBTFuTNUF9/QtIb4//33M1FAFroIjx5DCvC6PNnukAgsRhJj0GU2wXZZL0mNQc9hmhAYTBKMvhGs2uHEMyN3rtxClTg1C6EA/Nc217oC4c52Vy0tkVcu4kCR3WQ0oxBJoBgnDp0FaQpn+8MhUDRJqDbo1CxTcbcxAlMRGEQSCGDaHWQgwYteAMJACOcUquxKYumIdEQ+pAER1KUj0kIfEfOCwOd/17KWiILT46QN+dkYgQSBTZCE8syMArKI22nl5rsR6IvAIJLQMg56BchCglf2Odf0df4C4S9SgDDQUYg0YiGVB+7kC2LgQu9BGK4uAwExm8DwTFxKuyus3Q+BwKZIQjWCrgLl92vt9JCD70agBwKDSAJiYMTNej13/rMsw8X/nG2Q+BjN6/wDwho7SKOOJBDq8o87ecIOYhkyK1A4YcfMhDT7kIzC+L5bBDZJEtQG+gpmFSjAt2o4UEg5fLBw2RocRBLLZm291JiBRANBQBTMoLZqIHBmWCa7STW4WZJQqTl/8fDhw+L1FezuYgZ0796903ZgtgTXXbdv375S1nsbsGo5790kMQBPZjMsQTFDKdkww2OZjXddSRlfd+e0POVh9pQSY8nlWzFvmycJYVeivgJhBIFBBgh/SAKyaNKpQAqQHi9CvLy8PIW7uLjw7i5Vcqa7SWIEkCx7laavYImNd1tBBgh/XmNCPpv0RMwqCMPqA2Uh3AcfnJf3ci4bjoC35CC7IQmBXIK+AiLg1SPXr18/Cfwpy0nExQsRIRrKZjMdAZPEBAxL0Fcwa4AUEPLkR5sJxhSLuEQY7PAyWbyH4u5IghIyIud8xdL6CmYId+7cOREEwj2nIW7NSniDrs14BEwS47E7hWSNH4GKgF7SQAYPHpzJAeV8TkPcmpWwFGVzhcAuSUKl06vIcwtsxR/vEBKjfZaK5jSUiRkKy1E24xAwSYzD7b1QS+or0Ikwc+AdV3Malqpu3aqqjz6ywvsNzrsmCbUlRt5znq8gbgiiSdegfOS8M1MiTSu3h6NqkhiOWWsI6Svqtum2BuzpyKwFgljy/Ab6jaXT7AnH0t4OQRIClRcK5j5fwW4ldAZrGM1eliSnNcqZO02TRG5E38TH+Q62zU7REaRZQ1+A/mENXYFmL+gtDmwORRLUc059xd27d096gjXbD5+BXXoWs2Z5c6RtksiBYkMcCHNG/jnOV0AQLPusaSAIZhRzzZLWLFvPtA9HEsJF5ytSfUXf5RuWmNaaQagMurOcBlFM2UWluI5wN0ksUMus7SPkx56vYLmHGUQJRkRx0HMVhyUJtb1UX8GrPlhCajMs86A8LsmwPZazGDbdCJgkujHK5mOMvoIwjN5LMux4Ki1PC+FTWE0sVOqaZNBX6It5zBCadg+x/l+qwpjtt5TDph0Bk0Q7PrO49tVXsL22VIUxL2BE53IwY5IIFc5yDSNynXZmSSk1jNaXPn+R5qHpP0tlpRJYU57XsDdJrIF6dVY+d+krEMQsNZVqSiWwGfEySbwBlxkCwl8H1hC2XJEoUBJzkrpkw0yiaRZUcr6XzJtJYkm0a9JCX8HBtfRAHMrh0pd0yHMpupIaaOewMkm8QZVROEpsZhLpS/hEFOghUkX3HJUyNU7IjW+F29QjYJKox2VxW15njnJbO4fYybSF084Q2YG2xZokGnoGCmx2QDEyZ6sreoqbN282+C7LmjyL2MrKWRm5MUmUUQ9XueCFezowd2VZ8AOzCd42exBjkuhZ0SxDlaqLSIsg3YQ/ypQic/5vkqjHZVVb9BBzv3IjZwGZTRxkS6xJomfDYQlnS+cQWDLziwDrK9ckUY/LqrYI3aZXfK+asYbE0amwY+sAxiTRo5LRQ7C9dEtGSvgt5XmpvJoklkK6Zzroz0pXWKdF4SzH2qfB0zzN9N8k0QNY3ve0tW856DxHj+IdzotJorAq1xmKpbKV4z1QOs+xVJ5zpcN7tbQ5gDJwIh4dSwsmJoke4KO4HrqrCf9smZVBP8DX83KYvroRlsj6vmYkR762EodJorCaYoeTBFfurKE3QMms14Owkwr9B9+lmGqY/WxxFyF4y4ADZNFiTBIt4MgJYTvkTavaFRV3GOWciSD4+xAFB//IS+kGAgUfEbHuc+XbJDEXsiPj5fsNc24pRSjqzbQ8o/vIkR4H/0Q+I4u+aDDNGuLHonrs0jJJdNQSAmyo0poPD0XhrIN6HUkNctYrRNoCcahuSeU15RxzPkNfEdSy3tx5Nkm0tZoV3BiRS4jnTF67j6I+MedBOF7R0aW8ZuYyR9mG4kRemTGwtKRzKcyC4un2hnyaJDrAFkl0eLtyhhxYnmKkr91QPEfhyTNLUdhzR0jWGYQuhINQw8T3MrF0FYmoLryEb53bXHaQF6TKnRkB+HUZ4UR5cy3JtaVpkmhDZwU3SKJHOxmUM85d0G94Zbnet4SAhDBQOucwbNmNQpY4KQfxkyblyvHK9Bx5ZdaDIV9aIoPgONCI4d6wu8wkcYao8RcBhtAbYtK3yLKUIkFIPNJVaMdU08gZf1xyh3xkEMCyl116R1CTNstTCOAlLtJimQ3MdHEQEbKLRJnmFcJbgiBI1ySRor/y/9wkgWCGEDAc1NOSkPQRuYpLOpARQpclHJZuKIsu3BidI3yXuDRzqisfxAURgAfkiV8ITktQT5/WhTrZDZN+jdHs12HoTAIy4OBdNCyjRJLArc5fDKNnvYcJAR+XmBB0XUKVmQQCG3/MWpa4SCslCcgComAmVKdIpyxRD9E1QxI2Y+8mibHIzRQuN0kQnwwjaC2jMHIWech9yh3BqvMSKUGQB0btpIdgXuISGaZlYnYDBlwsObU9p2Grqgpo1rja6rRcMmQmgYBLldQIR4Q8BnJASDFDQGhjJCAJm5IJwhWDX/njP+Hj/5On5GdpnQTJkyfw4hUm4EBZ2wxlhlQJB4EQxiTRhtgO3RCoHTtsBpU6kgTLSwhFRs4I9IYllUHxyzMzhagExp7ROvYffHCeUWj9X2HWuFN2do9BWLo0g9B/7g0kY5LoUWkIvboRcF3QVJjjB6GHUMdAFozw+c8SDEtGWrdnxpHODiAYBC1CV/6IB2GaEsopgfDDclaXkA7eJz9SNsrUFysSxD+GcoJz1xLa5Ex6uSkHhHnjQKDmFKboChDWCD5m9dpeGxXYOUpA3KTRZCAkSASSKsmAdZfCPeTXJBHAaHpkK2mXsEVgMxpuUkLHpaKmdLCPswMEJulCDnE7Lf7S2UpdnAhdzWDq3I9qB6bSBx0Vg6LKPcdW0qgIR0jzX8rbXIXPTW658tUVDyQh4uzy6+WmHghV1Unwa8TbFILRs2YIdX6YTUQCqPOTEhH+uRhlR8NspWsWQX4gCZv3ETBJvI/JqjZSAM+VCWYQpJFztgLp0L8iGc2V/1zxMrtiSYkZkJTc0qng1mAsRRqAidaM6KUbiPZDn3OM6plV9FnOyZXnoWXcgn+TRGG1hN4q5/mFtHjoO3LqIoifPHMIcCsGMtOGGohBhllWC0HgzSQhsFrujNq3Nirf0qvNW6CfxckkMQus0yJlVF7a2n1bidg+uqVXmzNzIL8QJruumFVBnNixhbdlg4lJoq0hBDfOKEj5HKyLfYTU+sw4ii3AjBkzScwI7tioGeUOUKaOTSZbOEhNh9KyRTpzRBADF4QAWZB/iKJN+e6ZRP9KgSDiYbb+IZf3yQ6p9EDf8rkoN0WTRIF1g/BCEbwFw/LMlpaaJmLqmUQLgOgQdDjs4uLitOTUdmq4JapFnW7cuNG5G2vRDBWWmEmisAphNKvXWHSsjxeRcxThHaPvIvKZKRMmiQAkO5DYOVR3YpilG9xKH6GjsL51oFFOqL7ejyaJ3lDN6xFlKuTAziMMO29Kb7ss1Rxs16BJ4k03YNcQ2005f8CyEmv6uuKuJOy6trK+iXKVG/mb+8TyKgXLmKhJIiOYY6PikBkEkZ60ZpfTgD38Y5MfHQ6CaDiZPDrOwgOaJJIKYiTOwTcRRLq8hIDBrUTDDCg9dFdiPtfOk0lixRrQPv0mpS/29K+UPFbM8lXSvKvpIJ8svSqzFddvoWD0zbZRBAjEABGkBCHfHJrTi/dkt/YdxXqp5LU2Nmn6JokUkQX+I/zZwdTnNd0sP815bmJMcVGsl0peY8ozIEyZQ+IBBZjqFV0DQh99gwxLTgiSNsO6f59XY7TFkcuti9RypbOXeEwSC9ak9uWnL8LrygJvVY2Hvrr8z+nOGQMIokMmzJmFNeM+NEkg5Lnii/OojKiDaKocHbDrel1HU/hc9uSVGcSWznDkKvvYeEwSY5EbGK5J79A3GrbEph/16Rs2lz8O+EEQYRCZK+qtxHNIktC21j5k0FaRzEIQ0GvNKDSDQI9i0x8Bk0R/rEb57NI79I2U3U8QBQruNYxmEENnQWvkdcY0D0USUe+QC1NmFJxLSD80lCv+pnggBgiq5J1WTXlf294kMVMNDNE7DMkC5xJYflrScF6DGcSBzkM0wXsIkmDEz66fqHdoAmSsPSSB0G5Sdo+NNw3H0pjSmjoTSuM+yn+TROaaHqt3GJKNJQ/bKS2U1Tb7f8Gf9A5L1LV2GEFIEFNuA8lBRE3frMid3l7jM0lkrFl2IiFUl3hltg6yPXjw9lXXGYtymjUwe9DbUnPGveG4djuTQKDOJay76psvzCHMSX/qwTaWs/TVNk57e/bQhX63u0miG6NOH9I75H4Fd2fC1fmENsKcjwhN3XEEuaGUJj623Xr28F4N7I4kEAAsx0wVzu8hNdAiCncIA+JAf9A1wyAcS1bsmtLJb8hm7mWsgcXbtHeTxITqQ4j2Pe8wIZleQXlrLIptBDyzGc5g6GM6TRFACuhOIAb0HCIbk0MTYvtZbmKEjTBNvxHdWPIFHRDwLHvxyUwIg4tvVjMz4FAed71AUG5tX7lbMOu7TMokMaJa2QqKIC7xdd4IfZa9OA2N0OeCPPjP0hRkgPI7unEGg9nQEstkI+AuKcjmZxIocpfUO+SoPGYTzHSYWaDH4EJwQXTpmY0c6TmOdxEwSbyLR+c/BPCXX5b5qoy6zKNIZ2bADiVmF1w8szRmUqhDrNVu0ySB3oERd9cSTisCdjwcAiaJnlW+pt6hZxbtbX4ENkkSdPIS9A7zV49TmAOBrCTByFSj0/icK+MxzvjcFP+YF+OxXBPPA5Skd2gqp+0XQ2BTJFGy3mGxGnNCkxHIShIsY2idHmUo3wwWaUzOaXVeJhkSPzt+hhidKibPOu+g9IbEY7+7RWAzJMHuoLVef7Hb2j9owbKRBAQBKWgbKM9du2uGYD4kfmYDbAfVu44Ii53yVpeuCAKlLsRA2JwEV5em7TaHQPEkseZ5h83VpjPcC4EsJIEiF8OOH5k4imc30BSj+OObUOMhr7isBBHwfiHuWjbSjp8moa8X12nHDwSRk+CmlN1hi0JgFZLgvUNsV8Www6fuBXXYW+9QVFvZTWaykIQENsJcwlUCnTV9hPQUI/LpEz+zABGFDpex9AUR1OUjJQgRBdtFS/4q3BQ8HXY0AouTBDuRODB2//79U6b5Clw8RYw7BFJHHKNL6YBGICCQhSQgAi52ADGqZ5slo3juCOg6wxKQtmTqTniRTAwT42c2gL+m+HFXfISDMCCHureX4qZzBJAapICdjRFoQGBxkiAfzBJYRsKILJQ/SMJnBYSG73MgMJkkEOoI7bYLoZ0aRvB1V7okNDT+tnxALDF+/JoU0prx/xYEViEJCIKOqt1KLfmzkxHIjsBkkkDoRmHPqD3+13OacwQ2I/d4MVhKhXYaP+4Qh+LVXfHrP3f8xv882xiBCQisQhLMFDhlzG4lLytNqD0HHYXAZJJIU5V+IrXP9Z9loTgbaIs3Ks/b/NnNCPREYBWS4B1GGCmve+bV3oxAFgSykQSjdvQM2onEUg4GvUAOw+yB+KXEJl6WseqU0bjhl1doSEdCHur85sib4zgMAquQBC+8YyZhYwTWQCALSSDAnz49C2TtLtIZBQn1KYVj5iDyUbyQUt3yFDuaWMqCJHjGn8LmyMuUcjjs5hFYhSQ2j5oLsGkEspAEQpnRO3fNICSYIY+pBmFPvAh+0sBAEJBTOlPhEB8GMmHmAMEwo8Cf8nb24V8jMBgBk8RgyBxg6whkIQlAYOkHIc6l/wjqVIifXYf/Ej9kwV1KaS0nxdi0vEQ+SBuSIAy6knjoLobxsxHoiYBJoidQ9rYfBLKRRISEJR52LZUwco+zkJhHPxuBEQiYJEaA5iDbRmAWktg2JM69EWhEwCTRCI0d9oqASWKvNetyzYGASWIOVB1n0QiYJIquHmeuMARMEoVViLMzPwImifkxdgr7QcAksZ+6dEl6ImCS6AmUvRmBqqpMEm4Gh0PAJHG4KneBJyBgkpgAnoNuEwGTxDbrzbleBwGTxDq4O9UVETBJrAi+k94cAiaJzVWZMzwVAZPEVAQd/kgImCSOVNsu6wkBk4QbghHoj4BJoj9W9rkTBEwSO6lIF2MRBEwSi8DsREpCwCRRUm04L6UjYJIovYacv+wImCSyQ+oId4yASWLHleui1SNgkqjHxbZGoA4Bk0QdKrbbNQImiV1XrwuXGQGTRGZAHV35CJgkyq8j57AcBEwS5dSFc7IQAiaJhYB2MrtAwCSxi2p0IYYg8PLly+ry8nJIEPs1AodFwCRx2Kp3wY2AETAC3QiYJLoxsg8jYASMwGERMEkctupdcCNgBIxANwImiW6M7MMIGAEjcFgETBKHrXoX3AgYASPQjcD/AVtFFjP7KY3WAAAAAElFTkSuQmCC)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jpcOesiYncDY"
      },
      "source": [
        "# Gradient Descent "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 309
        },
        "id": "QbGbbXfcoOeR",
        "outputId": "d3b4cb7d-7484-47f6-fc20-04d6431f0359"
      },
      "source": [
        "from IPython.display import Image\n",
        "Image(url=\"https://thumbs.gfycat.com/AngryInconsequentialDiplodocus-size_restricted.gif\")"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"https://thumbs.gfycat.com/AngryInconsequentialDiplodocus-size_restricted.gif\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {},
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V8sDDWxwn0Cb"
      },
      "source": [
        "We usually update the weights using gradient descent algorithm. It tries to find the minima of the loss function so that the model is optimized. The formula to update the weights using gradient descent is\n",
        "\n",
        "w = w - (lr) * (w.gradients)\n",
        "\n",
        "Gradient descent is one of the many optimizers that can be used. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nqnglkFJ8Cj0"
      },
      "source": [
        "# Autograd\n",
        "\n",
        "Autograd is used to calulate the gradients of a tensor. It is a very useful tool when we know that we would require to calulcate the gradients of a tensor. It stores a computational graph of the tensor. It simply works by setting requires_grad=True\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4D4DCRgLLnq6"
      },
      "source": [
        "import torch"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DAWct_lLSvm6",
        "outputId": "0ba5aff0-6e7e-4aab-9adc-ca4ab9cef9a7"
      },
      "source": [
        "x=torch.randn(3,2,requires_grad=True)\n",
        "print(x)\n",
        "\n",
        "# In the output, we can see the AddBackward, i.e. the output has been obtained by adding\n",
        "y = x + 10\n",
        "print(y)\n",
        "\n",
        "# In the output, we can see the MulBackward, i.e. the output has been obtained by mutiplication\n",
        "z = y*y*y\n",
        "print(z)\n",
        "\n",
        "# In the output, we can see the MeanBackward, i.e. the output has been obtained by taking average\n",
        "w = z.mean()\n",
        "print(w)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[-1.2951,  1.6794],\n",
            "        [-0.4951,  1.5315],\n",
            "        [-0.5523, -0.4757]], requires_grad=True)\n",
            "tensor([[ 8.7049, 11.6794],\n",
            "        [ 9.5049, 11.5315],\n",
            "        [ 9.4477,  9.5243]], grad_fn=<AddBackward0>)\n",
            "tensor([[ 659.6056, 1593.1687],\n",
            "        [ 858.7086, 1533.4122],\n",
            "        [ 843.2810,  863.9839]], grad_fn=<MulBackward0>)\n",
            "tensor(1058.6934, grad_fn=<MeanBackward0>)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VFq3ZEw5T4H2"
      },
      "source": [
        "To see the gradients of the tensor, we just need to call .backward() method. The gradients are calculated with respect to the original tensor and are stored in original_tensor.grads"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HPo2BBITT0hU",
        "outputId": "bb62aecf-8b2b-48c7-8a56-9b8106732ccd"
      },
      "source": [
        "x=torch.randn(2,4,requires_grad=True)\n",
        "print(x)\n",
        "\n",
        "y=x+2\n",
        "print(y)\n",
        "\n",
        "z=y.mean()\n",
        "print(z)\n",
        "\n",
        "z.backward()\n",
        "print(x.grad)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[-0.6995, -0.8190, -0.0873, -1.0672],\n",
            "        [-0.0316, -0.7473, -0.3984,  1.0585]], requires_grad=True)\n",
            "tensor([[1.3005, 1.1810, 1.9127, 0.9328],\n",
            "        [1.9684, 1.2527, 1.6016, 3.0585]], grad_fn=<AddBackward0>)\n",
            "tensor(1.6510, grad_fn=<MeanBackward0>)\n",
            "tensor([[0.1250, 0.1250, 0.1250, 0.1250],\n",
            "        [0.1250, 0.1250, 0.1250, 0.1250]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RnO4CD_TWPhE"
      },
      "source": [
        "Note that the last value was a single valued tensor i.e. a sclar(as we computed mean). So there was no need of specifying the vector with respect to whose gradient we needed to calculate. However if the last value would have been a vector, we need to pass a vector of the same dimension as that of the last value to the .grad() function in order for pytorch to know in respect to which values of the vector, it needs to calculate the gradients."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fWITyeo2TFCh",
        "outputId": "6ac44797-8b7f-4213-ee28-dd693720ada5"
      },
      "source": [
        "x=torch.randn(2,4,requires_grad=True)\n",
        "print(x)\n",
        "\n",
        "y=x+2\n",
        "print(y)\n",
        "\n",
        "z=y*y\n",
        "print(z)\n",
        "\n",
        "w = torch.randn(2,4)\n",
        "z.backward(w)\n",
        "print(x.grad)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[ 0.2736, -1.7021,  1.0001, -0.5356],\n",
            "        [ 0.3485,  2.0273,  0.1088, -1.2772]], requires_grad=True)\n",
            "tensor([[2.2736, 0.2979, 3.0001, 1.4644],\n",
            "        [2.3485, 4.0273, 2.1088, 0.7228]], grad_fn=<AddBackward0>)\n",
            "tensor([[ 5.1691,  0.0888,  9.0008,  2.1444],\n",
            "        [ 5.5153, 16.2189,  4.4470,  0.5224]], grad_fn=<MulBackward0>)\n",
            "tensor([[ 6.8855,  0.7729,  6.8181, -1.7680],\n",
            "        [ 1.6124,  1.8710,  2.1220, -0.3147]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZfJlN5tVXhNd"
      },
      "source": [
        "Sometimes, we do not require pytorch to track the graidents, so for such times, we can either directly set the requires_grad to false or use x.detach() or wrap the functions in 'with torch.no_grad():'"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OVZF7aJhWOU7",
        "outputId": "999cc65e-36de-4ef7-97fa-2209f3c3bc1d"
      },
      "source": [
        "x=torch.randn(2,4,requires_grad=True)\n",
        "print(x)\n",
        "\n",
        "# As we can see, we do not have the grad_fn in the y and z tensors\n",
        "y = x.detach()\n",
        "print(y)\n",
        "\n",
        "with torch.no_grad():\n",
        "  z = x+1\n",
        "  print(z)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[ 0.0586, -1.7957, -1.6620, -0.1025],\n",
            "        [ 0.4816, -0.1239, -0.2575, -0.2463]], requires_grad=True)\n",
            "tensor([[ 0.0586, -1.7957, -1.6620, -0.1025],\n",
            "        [ 0.4816, -0.1239, -0.2575, -0.2463]])\n",
            "tensor([[ 1.0586, -0.7957, -0.6620,  0.8975],\n",
            "        [ 1.4816,  0.8761,  0.7425,  0.7537]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JXT42aO0ZLbG"
      },
      "source": [
        "Many a times during training, we need to flush out the gradients, so that they are not accumulated again and again during other epochs. This is achieved simply by using the tensor.grad.zero_() method"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jb6cW1HfYGte",
        "outputId": "976d5999-95fe-48a9-aabb-d4246649b26f"
      },
      "source": [
        "x=torch.randn(2,4,requires_grad=True)\n",
        "print(x)\n",
        "\n",
        "for i in range(3):\n",
        "  y = (x+2).mean()\n",
        "  y.backward()\n",
        "  print(x.grad)\n",
        "\n",
        "print(\"The above adds the gradients again and again and hence is incorrect. The correct one is shown below \")\n",
        "\n",
        "for i in range(3):\n",
        "  y = (x+2).mean()\n",
        "  y.backward()\n",
        "  print(x.grad)\n",
        "  x.grad.zero_()     # This flushes out the gradients"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[ 1.1620,  0.8418, -0.8677,  0.2599],\n",
            "        [-0.9022,  0.8239, -0.5743,  0.1103]], requires_grad=True)\n",
            "tensor([[0.1250, 0.1250, 0.1250, 0.1250],\n",
            "        [0.1250, 0.1250, 0.1250, 0.1250]])\n",
            "tensor([[0.2500, 0.2500, 0.2500, 0.2500],\n",
            "        [0.2500, 0.2500, 0.2500, 0.2500]])\n",
            "tensor([[0.3750, 0.3750, 0.3750, 0.3750],\n",
            "        [0.3750, 0.3750, 0.3750, 0.3750]])\n",
            "The above adds the gradients again and again and hence is incorrect. The correct one is shown below \n",
            "tensor([[0.5000, 0.5000, 0.5000, 0.5000],\n",
            "        [0.5000, 0.5000, 0.5000, 0.5000]])\n",
            "tensor([[0.1250, 0.1250, 0.1250, 0.1250],\n",
            "        [0.1250, 0.1250, 0.1250, 0.1250]])\n",
            "tensor([[0.1250, 0.1250, 0.1250, 0.1250],\n",
            "        [0.1250, 0.1250, 0.1250, 0.1250]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OjgCePcibb-L"
      },
      "source": [
        "Having learnt this, now let us implement a small linear regression in pytorch with the recently learnt autograds."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PiIPoaaYh2Wg"
      },
      "source": [
        "Here's an overview of the linear regression algorithm\n",
        "1. We assume the hypothesis function as y_pred = w * x + c\n",
        "\n",
        "2. Loss is calculated using y_predicted and y_true. \n",
        "\n",
        "3. The derivative of loss wrt the weights is computed. \n",
        "\n",
        "4. Weights are updated using gradient descent and we again go to step one."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IkOdjvidZy-9",
        "outputId": "db50054c-2cf9-4479-9969-0f451afa20b7"
      },
      "source": [
        "# We import nn for the loss function.\n",
        "import torch.nn as nn\n",
        "\n",
        "# Our data points, here we have only two :( \n",
        "x = torch.tensor([10.0,5.0])\n",
        "y = torch.tensor([10.0,5.0])\n",
        "\n",
        "# Initialize w and b as 1 and 0 respectively\n",
        "w = torch.ones(1,requires_grad=True)\n",
        "b = torch.ones(1,requires_grad=True)\n",
        "\n",
        "# Define the forward pass\n",
        "def forward(x):\n",
        "  return w*x+b\n",
        "\n",
        "# Define hyperparameters i.e. learning rate and epochs\n",
        "epochs = 100\n",
        "learning_rate = 0.001\n",
        "\n",
        "# Build the linear regression loop and train it for sepcified epochs. We use simple Mean squared loss (MSE)\n",
        "for i in range (epochs+1):\n",
        "\n",
        "  # Forward Pass\n",
        "  y_predicted = forward(x)\n",
        "\n",
        "  # caluclating Loss\n",
        "  loss = nn.MSELoss()\n",
        "  L = loss(y,y_predicted)\n",
        "\n",
        "  # Calculating gradients for the loss (Backward pass or backpropagation)\n",
        "  L.backward()\n",
        "\n",
        "  # Manually Updating weights starts here\n",
        "  # Since we do not need pytorch to track this updates, as this is not used in backprop, we use it inside no_grad()\n",
        "  # with torch.no_grad():\n",
        "    # w -= learning_rate*(w.grad)\n",
        "    # b -= learning_rate*(b.grad)\n",
        "\n",
        "  # WARNING: Do not forget to flush out weights\n",
        "  # w.grad.zero_()\n",
        "  # b.grad.zero_()\n",
        "  # Manually updating weights end here\n",
        "\n",
        "  # Instead of manually updating the weights, we can use optimizer present in pytorch.nn module\n",
        "  # Here, we have used SGD(Stochastic Gradient Descent)\n",
        "  optimizer = torch.optim.SGD([w,b],lr=learning_rate)\n",
        "  optimizer.step()\n",
        "  optimizer.zero_grad()\n",
        "\n",
        "  if (i%10 == 0 ):\n",
        "    print(\"Epoch \" + str(i))\n",
        "    print(\"Loss \" + str(L))\n",
        "\n",
        "print(y_predicted)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 0\n",
            "Loss tensor(1., grad_fn=<MseLossBackward0>)\n",
            "Epoch 10\n",
            "Loss tensor(0.1568, grad_fn=<MseLossBackward0>)\n",
            "Epoch 20\n",
            "Loss tensor(0.1004, grad_fn=<MseLossBackward0>)\n",
            "Epoch 30\n",
            "Loss tensor(0.0963, grad_fn=<MseLossBackward0>)\n",
            "Epoch 40\n",
            "Loss tensor(0.0957, grad_fn=<MseLossBackward0>)\n",
            "Epoch 50\n",
            "Loss tensor(0.0953, grad_fn=<MseLossBackward0>)\n",
            "Epoch 60\n",
            "Loss tensor(0.0949, grad_fn=<MseLossBackward0>)\n",
            "Epoch 70\n",
            "Loss tensor(0.0945, grad_fn=<MseLossBackward0>)\n",
            "Epoch 80\n",
            "Loss tensor(0.0942, grad_fn=<MseLossBackward0>)\n",
            "Epoch 90\n",
            "Loss tensor(0.0938, grad_fn=<MseLossBackward0>)\n",
            "Epoch 100\n",
            "Loss tensor(0.0934, grad_fn=<MseLossBackward0>)\n",
            "tensor([9.8049, 5.3857], grad_fn=<AddBackward0>)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pyM2NdKQS-AV"
      },
      "source": [
        "A very important point to note in the above code is that while updating the w and b if we write w = w - lr * (w.grad), it won't work. This is becasue this statement will store it in a new 'w'. Hence we use w -= lr * (w.grad)"
      ]
    }
  ]
}