examples/03_meta_a2c.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J1b2VjZJ2i09"
      },
      "source": [
        "# Learning to Reinforcement Learn with `gymnax`\n",
        "### [Last Update: December 2022][![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/RobertTLange/gymnax/blob/main/examples/03_meta_a2c.ipynb)\n",
        "\n",
        "*Note*: This notebook was partially used to teach at the M2L summer school 2022 in Milano.\n",
        "\n",
        "In this final exercise we will implement a memory-based meta-reinforcement learning agent. But first what is meta-learning more generally? So far we have trained RL agents to perform a single task (CartPole, etc.) using general-purpose algorithms like temporal difference learning. The neural network executes a policy, which is tailored to the training environment. But what will happen if the environment dynamics change? The agent will most likely not be able to adapt given that the network weights are frozen.\n",
        "\n",
        "One approach to overcome this limitation is to instead train a network on a distribution of tasks and to explicitly provide the previous reward and action as an input. If the network is an RNN it will be able to adapt its behavior to the incoming information. This can ultimately lead to the discovery of learning algorithms encoded in the network dynamics of the RNN. Hence, by training with RL on many different tasks networks can **learn how to learn and adapt**. Note that there are many different meta-reinforcement learning algorithms and we will only focus on the case of memory-based meta RL simultaneously introduced by [Wang et al. (2016)](https://arxiv.org/abs/1611.05763?source=post_page---------------------------) and [Duan et al. (2016)](https://arxiv.org/abs/1611.02779).\n",
        "\n",
        "More specifically, we will consider a two arm bandit setting in which the best arm is sampled at the beginning of each episode. In order to perform in every episode the agent has to meta-learn an exploration algorithm, which integrates reward information from the different arms.\n",
        "\n",
        "Let's start by implementing this simple environment in plain JAX. JAX environments allow us to easily collect data from multiple actors in parallel: We can simply `vmap` over different random number keys. This is especially powerful for distributed actor-critic agents, which rely on efficient on-policy data throughput."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8iLoPxPRnxqy"
      },
      "source": [
        "# Installations"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "executionInfo": {
          "elapsed": 85,
          "status": "ok",
          "timestamp": 1660492684926,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "5NhpFfT5ntO4"
      },
      "outputs": [],
      "source": [
        "!pip install --quiet dm-haiku optax distrax chex gymnax\n",
        "\n",
        "from IPython.display import clear_output\n",
        "clear_output()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "executionInfo": {
          "elapsed": 9701,
          "status": "ok",
          "timestamp": 1660492694721,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "A_dvUXyXKsu1"
      },
      "outputs": [],
      "source": [
        "from typing import NamedTuple, Tuple, Optional\n",
        "\n",
        "import chex\n",
        "import distrax\n",
        "import haiku as hk\n",
        "import jax\n",
        "import jax.numpy as jnp\n",
        "import matplotlib.pyplot as plt\n",
        "import optax\n",
        "import gymnax"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bssr_gDAGO2M"
      },
      "source": [
        "# Define the Bandit Environment"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "executionInfo": {
          "elapsed": 104,
          "status": "ok",
          "timestamp": 1660492694917,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "KNCuLFTfPfiu"
      },
      "outputs": [],
      "source": [
        "env = gymnax.make('BernoulliBandit-misc')\n",
        "env_params = env.default_params"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fD9PnHKGGT5l"
      },
      "source": [
        "Note that the bandit observation includes the reward $r_{t}$, one-hot encoded action $a_{t}$ as well as a normalized timestep counter $t$. This is ultimately all the information needed for an RNN-based agent to learn which arm is better in a given episode. The parameter `reward_prob` sets the task difficulty. Intuitively, small values will lead to arms with very different expected pull value. Arms which are closer together will be harder to tell apart.\n",
        "\n",
        "# Define the Meta-Reinforcement Learner LSTM\n",
        "\n",
        "Next, we will setup a sample actor-critic network with an LSTM module. It is a fairly small model which process the observation inputs and has two output heads: One output policy logits for $\\pi(a | h_t)$ and the other outputs a value estimate $v(h_t)$, where $h_t$ denotes the hidden state of the LSTM."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "executionInfo": {
          "elapsed": 53,
          "status": "ok",
          "timestamp": 1660492695058,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "aI-9ycC4Rl3u"
      },
      "outputs": [],
      "source": [
        "class LSTMMetaRL(hk.RNNCore):\n",
        "  \"\"\"Simple LSTM Wrapper with flexible output head.\"\"\"\n",
        "\n",
        "  def __init__(\n",
        "      self,\n",
        "      num_hidden_units: int = 32,\n",
        "      num_output_units: int = 2) -> None:\n",
        "    super().__init__(name=\"LSTMMetaRL\")\n",
        "    self.num_hidden_units = num_hidden_units\n",
        "    self.num_output_units = num_output_units\n",
        "\n",
        "    self._lstm_core = hk.LSTM(self.num_hidden_units)\n",
        "    self._policy_head = hk.Linear(self.num_output_units)\n",
        "    self._value_head = hk.Linear(1)\n",
        "\n",
        "  def initial_state(self, batch_size: int) -> hk.LSTMState:\n",
        "    return self._lstm_core.initial_state(batch_size)\n",
        "\n",
        "  def __call__(\n",
        "      self,\n",
        "      x: chex.Array,\n",
        "      state: hk.LSTMState) -> Tuple[hk.LSTMState, chex.Array, chex.Array]:\n",
        "    output, next_state = self._lstm_core(x, state)\n",
        "    policy_logits = self._policy_head(output)\n",
        "    value = self._value_head(output)\n",
        "\n",
        "    return next_state, policy_logits, value"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lnjb5w3n-2vN"
      },
      "source": [
        "Instantiate the environment, reset an episode and perform a step. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "executionInfo": {
          "elapsed": 3730,
          "status": "ok",
          "timestamp": 1660492698883,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "Gtq2FByMRFf-",
        "outputId": "63fb95ce-7df8-4820-95eb-a2dd502eb165"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(DeviceArray([ 1.  ,  0.  ,  1.  , -0.98], dtype=float32),\n",
              " EnvState(last_action=1, last_reward=DeviceArray(1, dtype=int32), exp_reward_best=DeviceArray(0.9, dtype=float32), reward_probs=DeviceArray([0.1, 0.9], dtype=float32), time=1.0),\n",
              " DeviceArray(1, dtype=int32),\n",
              " False,\n",
              " {'regret': DeviceArray(0., dtype=float32)})"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "rng = jax.random.PRNGKey(0)\n",
        "obs, state = env.reset(rng, env_params)\n",
        "action = 1\n",
        "env.step(rng, state, action, env_params)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kTqDMYYWOUXA"
      },
      "source": [
        "# Define A2C Components\n",
        "\n",
        "In the following the network will be trained using A2C [(Mnih et al., 2016)](http://proceedings.mlr.press/v48/mniha16.pdf). A2C collects data of different workers synchronously (e.g. using a vmapped rollout function) and computes the following objective function:\n",
        "\n",
        "$$\\mathcal{L} = \\mathbb{E}_\\pi [\\log \\pi(a | h_t) [R_t - V(h_t)]^\\dagger] + \\beta_v \\mathbb{E}_\\pi [(R_t^\\dagger - V(h_t))^2] + \\beta_e \\mathbb{E}_\\pi [\\mathcal{H}(\\pi(a | h_t))],$$\n",
        "\n",
        "where $\\dagger$ denotes stop gradient operation and $R_t = \\sum_{i=0}^{T-t-1} \\gamma^i r_{t+i}$ is the return to go. It consists of three components:\n",
        "\n",
        "1. **REINFORCE objective**: $\\mathbb{E}_\\pi [\\log \\pi(a | h_t) [R_t - V(h_t)]^\\dagger]$. Intuitively, this objective tries to increase the probability of actions that lead to a higher than expected return outcome.\n",
        "2. **Value estimation**: $\\mathbb{E}_\\pi [(R_t^\\dagger - V(h_t))^2]$. This term tries to improve the capability of the critic to predict the return of following the agents policy.\n",
        "3. **Entropy regularizer**: $\\mathbb{E}_\\pi [\\mathcal{H}(\\pi(a | h_t))]$. The entropy measures the randomness of the policy and can be used to regularize the policy to be more or less exploratory. This plays a key role in discovering learning algorithms or the meta-exploration problem.\n",
        "\n",
        "Let us now implement a function to rollout the agent and collect the required statistics for computing the objective. We will use `hk.scan` (which is `haiku`'s equivalent of `jax.lax.scan`) in order to ensure that we can jit compile through the episode rollout in a fast manner."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "executionInfo": {
          "elapsed": 52,
          "status": "ok",
          "timestamp": 1660492699027,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "2QXT8mIWRjOn"
      },
      "outputs": [],
      "source": [
        "def compute_returns(rewards, gamma=1.):\n",
        "  \"\"\"Compute list of returns up to T.\"\"\"\n",
        "  R = 0\n",
        "  returns = jnp.zeros(rewards.shape[0])\n",
        "  counter = rewards.shape[0] - 1\n",
        "  for step in reversed(range(len(rewards))):\n",
        "    R = rewards[step] + gamma * R\n",
        "    returns = returns.at[counter].set(R)\n",
        "    counter -= 1\n",
        "  return returns\n",
        "\n",
        "\n",
        "def rollout(rng_input, gamma, env_params, steps_in_episode):\n",
        "  \"\"\"Rollout a jitted gymnax episode with lax.scan.\"\"\"\n",
        "  # Reset the environment\n",
        "  rng_reset, rng_episode = jax.random.split(rng_input)\n",
        "  obs, env_state = env.reset(rng_reset, env_params)\n",
        "\n",
        "  lstm = LSTMMetaRL(48, 2)\n",
        "  net_state = lstm.initial_state(None)\n",
        "\n",
        "  def policy_step(state_input, tmp):\n",
        "      \"\"\"lax.scan compatible step transition in jax env.\"\"\"\n",
        "      obs, env_state, net_state, rng = state_input\n",
        "      rng, rng_step, rng_act = jax.random.split(rng, 3)\n",
        "      net_state, logits, value = lstm(obs, net_state)\n",
        "      policy = distrax.Categorical(logits=logits)\n",
        "      entropy = policy.entropy()\n",
        "      action, log_prob = policy.sample_and_log_prob(seed=rng_act, sample_shape=())\n",
        "      next_obs, next_env_state, reward, done, info = env.step(\n",
        "          rng_step, env_state, action, env_params\n",
        "      )\n",
        "      carry = [next_obs, next_env_state, net_state, rng]\n",
        "      return carry, [log_prob, value, entropy, reward, info[\"regret\"]]\n",
        "\n",
        "  # Scan over episode step loop\n",
        "  _, scan_out = hk.scan(\n",
        "      policy_step,\n",
        "      [obs, env_state, net_state, rng_episode],\n",
        "      (),\n",
        "      steps_in_episode\n",
        "  )\n",
        "  # Unpack episode statistics\n",
        "  log_probs, values, entropies, rewards, regrets = scan_out\n",
        "  returns = compute_returns(rewards, gamma)\n",
        "  advantages = returns - values\n",
        "  return log_probs, advantages, entropies, jnp.sum(rewards), jnp.sum(regrets)\n",
        "\n",
        "\n",
        "# Batch rollout helper - vmap over rngs\n",
        "rollout_fn = hk.without_apply_rng(hk.transform(rollout))\n",
        "batch_rollout_fn = jax.vmap(rollout_fn.apply, in_axes=(None, 0, None, None, None))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "executionInfo": {
          "elapsed": 6023,
          "status": "ok",
          "timestamp": 1660492705160,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "jCjoHtiSGRjH"
      },
      "outputs": [],
      "source": [
        "# Rollout an episode and return collected stats\n",
        "env = BernoulliBandit()\n",
        "env_params = env.default_params\n",
        "\n",
        "rng = jax.random.PRNGKey(0)\n",
        "net_params = rollout_fn.init(jax.random.PRNGKey(0), rng, 0.8, env_params, 100)\n",
        "log_probs, advantages, entropies, reward, regrets = rollout_fn.apply(net_params, rng, 0.8, env_params, 100)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3J-RHPl0GSRL"
      },
      "source": [
        "Next up implement the A2C objective function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "executionInfo": {
          "elapsed": 53,
          "status": "ok",
          "timestamp": 1660492705308,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "WDHOX6L4A585"
      },
      "outputs": [],
      "source": [
        "def a2c_loss(log_probs, advantages, entropies, entropy_coeff, value_coeff):\n",
        "  \"\"\"Compute the actor-critic REINFORCE loss with entropy regularization.\"\"\"\n",
        "  # ============ YOUR CODE HERE =============\n",
        "  actor_loss = -(log_probs * jax.lax.stop_gradient(advantages)).mean()\n",
        "  critic_loss = (advantages**2).mean()\n",
        "  entropy_loss = entropies.mean()\n",
        "  ac_loss = actor_loss + value_coeff*critic_loss - entropy_coeff*entropy_loss\n",
        "  return ac_loss\n",
        "\n",
        "\n",
        "@jax.jit\n",
        "def batch_a2c_loss(net_params, rng_batch, gamma, entropy_coeff, value_coeff, env_params):\n",
        "  \"\"\"Batch forward pass for multiple actors & compute average A2C loss.\"\"\"\n",
        "  batch_loss_fn = jax.vmap(a2c_loss, in_axes=(0, 0, 0, None, None))\n",
        "  steps_in_episode = 100\n",
        "  log_probs, advantages, entropies, rewards, regrets = batch_rollout_fn(net_params, rng_batch, gamma, env_params, steps_in_episode)\n",
        "  batch_loss = batch_loss_fn(log_probs, advantages, entropies, entropy_coeff, value_coeff)\n",
        "  return batch_loss.mean(), [jnp.mean(rewards), jnp.mean(regrets)]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "executionInfo": {
          "elapsed": 352,
          "status": "ok",
          "timestamp": 1660492705746,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "rZvBYi-BZdLH",
        "outputId": "d5002f45-f367-4ed5-b13e-4b4be141a847"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "DeviceArray(5.632409, dtype=float32)"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Compute A2C loss for single episode/actor - for beta_v and beta_e\n",
        "a2c_loss(log_probs, advantages, entropies, 0.1, 0.5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "executionInfo": {
          "elapsed": 3405,
          "status": "ok",
          "timestamp": 1660492709238,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "oIbjj1TMSdp0",
        "outputId": "62dca86e-f741-496a-99bf-c9f05a53f7b9"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(DeviceArray(4.9391594, dtype=float32),\n",
              " [DeviceArray(51., dtype=float32), DeviceArray(39.2, dtype=float32)])"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Compute batch A2C loss for five parallel actors\n",
        "rng_batch = jax.random.split(rng, 5)\n",
        "batch_a2c_loss(net_params, rng_batch, 0.8, 0.1, 0.5, env_params)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XDFArOxLTfUq"
      },
      "source": [
        "# Run A2C Loop\n",
        "\n",
        "We are now ready to put everything together into a training loop. Our goal is to reproduce figure 2 f) of the [Wang et al. (2016)](https://arxiv.org/abs/1611.05763?source=post_page---------------------------) paper, which demonstrates the generalization capabilities of the meta-learned bandit RL algorithm to different bandit settings:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "executionInfo": {
          "elapsed": 51,
          "status": "ok",
          "timestamp": 1660492709378,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "XNPO6mpVZh02"
      },
      "outputs": [],
      "source": [
        "def run_a2c_training(rng, env_params, verbose):\n",
        "  num_updates = 4000\n",
        "  num_workers = 64\n",
        "  lr = 0.005\n",
        "  beta_ent = 1.\n",
        "  beta_ent_decay = 0.999\n",
        "  beta_value = 0.5\n",
        "  gamma = 0.8\n",
        "  num_ep_steps = 100\n",
        "  track_loss, track_rewards, track_regrets = [], [], []\n",
        "\n",
        "  net_params = rollout_fn.init(rng, rng, gamma, env_params, 100)\n",
        "  optimizer = optax.chain(\n",
        "        optax.clip_by_global_norm(5.0),\n",
        "        optax.scale_by_adam(eps=1e-4),\n",
        "        optax.scale(-0.005),\n",
        "    )\n",
        "  opt_state = optimizer.init(net_params)\n",
        "\n",
        "  for up in range(num_updates):\n",
        "    # Split random number keys for episode rollout\n",
        "    rng, rng_b = jax.random.split(rng)\n",
        "    rng_batch = jax.random.split(rng_b, num_workers)\n",
        "\n",
        "    # ============ YOUR CODE HERE =============\n",
        "    # Rollout batch episodes for workers via vmap & perform gradient update\n",
        "    out, grads = jax.value_and_grad(batch_a2c_loss, has_aux=True)(net_params, rng_batch, gamma, beta_ent, beta_value, env_params)\n",
        "    updates, opt_state = optimizer.update(grads, opt_state)\n",
        "    net_params = optax.apply_updates(net_params, updates)\n",
        "    \n",
        "    # Decay the exploration loss coefficient - meta-exploration\n",
        "    beta_ent = jnp.clip(beta_ent * beta_ent_decay, 0.01, 1.0)\n",
        "\n",
        "    # Track loss and print performance\n",
        "    track_loss.append(out[0])\n",
        "    track_rewards.append(out[1][0])\n",
        "    track_regrets.append(out[1][1])\n",
        "    if verbose and up % 200 == 0:\n",
        "      print(f\"# Updates: {up} | Loss: {out[0]} | Return {out[1][0]}  | Regret: {out[1][1]} | b_e: {beta_ent}\")\n",
        "  return net_params, track_loss, track_rewards, track_regrets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "executionInfo": {
          "elapsed": 404499,
          "status": "ok",
          "timestamp": 1660493113989,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "9TgrHgjHfPaR",
        "outputId": "63baedbb-5290-4887-809f-4e26da84d35b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "# Updates: 0 | Loss: 4.175103664398193 | Return 49.875  | Regret: 40.02499771118164 | b_e: 0.9990000128746033\n",
            "# Updates: 200 | Loss: -0.1845569610595703 | Return 65.359375  | Regret: 24.46249771118164 | b_e: 0.8178321719169617\n",
            "# Updates: 400 | Loss: -0.10990064591169357 | Return 70.546875  | Regret: 19.325000762939453 | b_e: 0.6695191860198975\n",
            "# Updates: 600 | Loss: -0.0999000146985054 | Return 71.4375  | Regret: 18.625 | b_e: 0.5481024384498596\n",
            "# Updates: 800 | Loss: -0.015548501163721085 | Return 76.53125  | Regret: 13.837498664855957 | b_e: 0.4487048387527466\n",
            "# Updates: 1000 | Loss: 0.040223054587841034 | Return 80.140625  | Regret: 9.749999046325684 | b_e: 0.3673325479030609\n",
            "# Updates: 1200 | Loss: 0.09893104434013367 | Return 83.03125  | Regret: 7.199999809265137 | b_e: 0.3007172644138336\n",
            "# Updates: 1400 | Loss: 0.1617896407842636 | Return 84.84375  | Regret: 4.474999904632568 | b_e: 0.2461823672056198\n",
            "# Updates: 1600 | Loss: 0.19916124641895294 | Return 86.6875  | Regret: 3.0 | b_e: 0.20153741538524628\n",
            "# Updates: 1800 | Loss: 0.24016234278678894 | Return 88.734375  | Regret: 1.6875 | b_e: 0.16498881578445435\n",
            "# Updates: 2000 | Loss: 0.24250157177448273 | Return 89.421875  | Regret: 1.2625000476837158 | b_e: 0.13506826758384705\n",
            "# Updates: 2200 | Loss: 0.27599838376045227 | Return 88.578125  | Regret: 0.9874999523162842 | b_e: 0.11057380586862564\n",
            "# Updates: 2400 | Loss: 0.2609182298183441 | Return 89.234375  | Regret: 0.8249999284744263 | b_e: 0.09052134305238724\n",
            "# Updates: 2600 | Loss: 0.26438429951667786 | Return 88.484375  | Regret: 0.8999999761581421 | b_e: 0.074105404317379\n",
            "# Updates: 2800 | Loss: 0.2833080291748047 | Return 89.3125  | Regret: 0.6374999284744263 | b_e: 0.060666490346193314\n",
            "# Updates: 3000 | Loss: 0.29172730445861816 | Return 88.8125  | Regret: 0.6874999403953552 | b_e: 0.0496646910905838\n",
            "# Updates: 3200 | Loss: 0.28429871797561646 | Return 89.15625  | Regret: 0.5624999403953552 | b_e: 0.040658049285411835\n",
            "# Updates: 3400 | Loss: 0.2838882803916931 | Return 89.40625  | Regret: 0.5 | b_e: 0.0332847461104393\n",
            "# Updates: 3600 | Loss: 0.2838461995124817 | Return 89.34375  | Regret: 0.5874999761581421 | b_e: 0.027248600497841835\n",
            "# Updates: 3800 | Loss: 0.28280776739120483 | Return 89.1875  | Regret: 0.5374999642372131 | b_e: 0.02230709046125412\n"
          ]
        }
      ],
      "source": [
        "net_params_10, loss_10, rewards_10, regrets_10 = run_a2c_training(rng, env_params, verbose=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "height": 299
        },
        "executionInfo": {
          "elapsed": 944,
          "status": "ok",
          "timestamp": 1660493115029,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "btTujaZTMg5A",
        "outputId": "2d793bbb-0899-40cf-e030-c2298ca33db7"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1YAAAEaCAYAAAASWcZVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABNRUlEQVR4nO3dd3gc1dXH8e9Rd69yLwJcwLiCMMUUgzHNEEooCQmBFEjeQChJSAykQAiJE2oKgUAIEGoooSRUY7BNN8aAsY0x7t2We1c97x87EpKsspJWO6vd3+d59OzuzJ2Zc1Wu5s5t5u6IiIiIiIhI46WFHYCIiIiIiEhLp4qViIiIiIhIE6liJSIiIiIi0kSqWImIiIiIiDSRKlYiIiIiIiJNpIqViIiIiIhIE6liJSIijWZmL5nZhTE+5/Vm9nAsz5lIzOwiM3sr7DhEkoHKIEkkqlilCDObamabzSy72varzWyOmW03syVmdnW1/WZmlwdpdprZSjN70syG1XGd7zVnXkQktsxsqZntNrMdlb7+Gs2x7n6yuz/Y3DFGq1pe1prZA2bWNuy4RKR2KoOaLRY3swFhXDtVqWKVAswsDzgKcOAr1XcD3wI6AScBl5nZ1yrt/xNwBXA50BkYBDwLTGjWoEUk3k5z97aVvi4LO6AmOM3d2wIjgVHANWEFYmYZYV1bpIVRGdQAKlsSkypWqeFbwHvAA0CV5nJ3/6O7z3L3Enf/HHgOGANgZgOBS4Gvu/vr7l7o7rvc/RF3n9SQAMwszcx+YWbLzGy9mf3LzDoE+3LM7GEz22hmW8zsAzPrHuy7yMwWV2pR+0ZTvxkiEr3gb/BtM/uLmW01s/lmNq7S/opWajMbYGbTgnQbzOzfldIdEfxtbw1ej6i0b5/guO1mNhnoWi2Gw8zsnaB8+MTMxkYTu7uvBV4hcnNT57nM7Fgz+7RSutfMbEalz2+Z2RnB+4lmtiiId56ZnVnD9+t2M9sEXG9mXczseTPbFpxzv2jiF5HUKYMqxTE9iOM1M7vTgi6JZpZnkRao75rZcuD1YPt3zOwzi/RKesXM+gfbpwen/cQirWfnRROzNI0qVqnhW8AjwdeJ5ZWW6szMiLRszQ02jQNWuvuMmtI30EXB17HAvkBboLyZ/0KgA9AX6AL8ANhtZm2APwMnu3s74Ajg4xjEIiINcyiwmMjNxq+B/5hZ5xrS3Qi8SqQFvA/wF4Ag7QtE/p67ALcBL5hZl+C4R4EPg/PfSKUHQGbWOzj2t0RazX8KPG1mufUFbWZ9gJOBhVGc611ggJl1tciT4KFAHzNrZ2atgIOBN4NTLyJSVnYAbgAeNrOeNXy/ugE3AXcCe4CewHeCLxGJXiqUQeVxzAhivB64oIbTHgMcQOR+7gzgWuAsIJdIGfUYgLsfHaQfEbQA/ruGc0mMqWKV5MzsSKA/8IS7f0jkhuD8WpJfT+R34v7gcxdgTYxC+QZwm7svdvcdRJrFvxbcwBQH1xrg7qXu/qG7bwuOKwOGmlkrd1/j7nNrPr2INNGzwRPU8q+LK+1bD9zh7sXBP+fPqbk7cDGR8qaXu+9x9/IJGiYAX7j7Q0Hr+GPAfOA0M+sHHAL8MmgVnw78t9I5vwm86O4vunuZu08GZgKn1JOX7cCKIPZf13cud98TvD8ayAdmA28RacE/LIh/I4C7P+nuq4Nz/Bv4Ahhd6fqr3f0v7l4CFAFfBX7l7jvdfQ6QMONBRBJISpdBleL4lbsXBbE/X8O5rw/Kkt3A94Hfu/tnQXnzO2BkeauVxJ8qVsnvQuBVd98QfH6Uat0BAczsMiItWxPcvTDYvJHIE9ZY6AUsq/R5GZABdAceItJU/riZrTazP5pZprvvBM4j0oK1xsxeMLP9YxSPiFR1hrt3rPR1b6V9q9zdK31eRuRvurqfERm3OcPM5ppZectM9b//8nP0DvZtDv7eK+8r1x84p/INF3AkdZdNZwSt3GOB/fmyW09955oWHHN08H4qkafDxwSfATCzb5nZx5XOMZSqXYdWVHqfS6Ssq7yt+vdCRFQG9QI2ufuuSuepXG7UtK0/8KdK59oU5L93HbFJM1LFKokF3VfOBY6xyMw0a4GrgBFmNqJSuu8AE4Fx7r6y0immEOkKkx+DcFYTKQDK9QNKgHXBE6gb3H0Ike5+pxKp5OHur7j7eCKFznzgXkQk3noHXYXL9SPyN12Fu69194vdvReRJ6l/s8iMVNX//svPsYpIq3inoOtv5X3lVgAPVbvhahPNOE93n0ZkbOktUZ6resVqGtUqVsGT4HuBy4Au7t4RmEPkZqbi0pXeFxAp6/rWkj8RqV8qlEFrgM5m1rrSKfqyt8rlywrg+9XO18rd36kvNmkeqlgltzOAUmAIkYGTI4n0y32ToOJikckgfgeMd/fFlQ929y+AvwGPmdlYM8uyyEQTXzOziXVcNyNIV/6VSaTP71XBwMy2wTX/7e4lFhk0PszM0oFtRJryS82su5l9JSjsCoEdQX5EJL66AZebWaaZnUOkHHmxeiIzOycYUwCwmcgNQGmQdpCZnW9mGcEg6iHA/9x9GZGuMDcEZcyRwGmVTvswke46J5pZelCmjK10nfrcAYw3s5FRnOsdYDCRbn0zgq7H/YmM7ygfCN4myFdBkOdvE2mxqpG7lwL/ITKJRWszG0INvQZEpE5JXwZViuP6II7Dq8VRk7uBa8zswCD/HYLvT7l1RMa1S5yoYpXcLgTud/flwVOctR6ZoeavwDcsMr7pt0TGN31gX64dcXelc1wepL8T2EJkjNaZVO1/XN1dwO5KX/cD/yTS5W86sITIQO4fBel7AE8RqVR9RuTJ8MNEfj9/QuRJ0yYiT41/2OjvhojU5b9WdQ2ZZyrtex8YCGwgMhnD2eXjjao5BHjfzHYQGRtwhbsvCdKeSuTveSOR7jqnVuqifD6RyssmImMR/lV+QndfAZxOZIB2AZEntFcT5f8vdy8IzvfL+s4VdAWaBcx196LgFO8Cy9x9fZBmHnBrsH0dMAx4u54wLiMyYc9aIk+v768ztUhqSvkyiMh49MODGH8L/JvIg+Xazv0M8AciQym2EWk9P7lSkuuBB4OugudGE680jVXtsioiIvIlM7sI+J67Hxl2LCKSelK5DLLIdPHz3f3X9SaWhKAWKxERERGRkJnZIWa2n0XW/jyJSOvWsyGHJQ2gVZtFRERERMLXg8iYzC7ASuD/3P2jcEOShlBXQBERERERkSZSV0AREREREZEmUsVKRERERESkiZpljFXXrl09Ly+vOU4tIgnmww8/3ODuuWHHoXJHJHUkSrkDKntEUkl9ZU+zVKzy8vKYOXNmc5xaRBKMmS0LOwZQuSOSShKl3AGVPSKppL6yR10BRUREREREmkgVKxERERERkSZSxUpERERERKSJVLESERERERFpIlWsREREREREmkgVKxERERERkSZSxUpERERERKSJQq1YPTlzBQ+9lzBLUYiIiCSMN78oYOH6HTE/757iUjbvLIr5eaVxtuwq4tmPVoUdhojEQKgVq//OXsPTH64MMwQRCZmZXWFmc8xsrpldGWzrbGaTzeyL4LVTyGGKVDF75RbcPaq0p/7lTb7/UMMXkL3gvhkcf9s0AN74fD1PfLCiweeoyXn3vMeoGyc36JiZSzexfU9xTK4vsHLzLr7/0EyWbdzJyN9M5sp/f8ySDTvDDktEmij0roDR/VsSkWRkZkOBi4HRwAjgVDMbCEwEprj7QGBK8FmkXnuKS5t0fHFpGSWlZXWmmfr5er7y17d5OMoeF3NWbeOVuev22v7Owg1sqqHlaOuuYu57a0mVbd++/wN+9vTsqK5Xn09WbGlQ+h2FJZx997v84OEP99o3Y8kmjr1lKruKShoVy4wlm3hyZmwqjImutMx5fMZyvvPAB/zsqdm8Mncdx9w8tWJ/cT2/dyKS+DLCvLgBRPnET0SS0gHAe+6+C8DMpgFnAqcDY4M0DwJTgZ+HEJ+0EO7O1M8L+PYDH/DspWMY2bcjOwtLyM5IIyO95meIq7fsJjsjjS5tsyu2DbzuJfbv0Y7//PAIrvnPp3Ruk8Wpw3sxtHd7PliymX1z2/DR8i0AzF+7vcr5ysqc426dylXjBwGwb9e2DOvToUqabXuKaZ+Tyazlmzn/H+8DMLh7O3531jBuemEeN3xlKHdPW8QLn66pMeY/vDyfXh1yaN8qk76dW7Ny825OG96TnUWlnHj7dO79Vj5DerVnzqqtPP7Bcm48fShmBsC81dtYt31Pxbme/2Q1g7u3w3FenrOWc/P70qtjKwDOv/c9zs3vy8Dubdm2O1JpmrNqG0UlZRz5h9dZv72QqT8dy7l/fxeA1+ev5+PlWygqLSPNjO7tc/i/sftV5PnONxbyv0/W8OB3RnP8bdP434+O5MBe7SuOPye/b10/3mZhZunATGCVu59qZp2BfwN5wFLgXHffHKvrffWud/i4jkqtxepCIhIai7YrQ0Pk5+f7zJn1d3u46P4ZbN5ZxHOXHRnzGEQkPszsQ3fPb+SxBwDPAYcDu4m0Ts0ELnD3jpXSbXb3OrsDRlvuSPJwd6Z8tp7hfTsw+qYpZKWnUVRaxq9PG8IR+3XlxDumc/SgXD5atpk7v3EQQ3t3oHObrIrj8ya+AMD1pw1hUPd2ZGem89W73on6+j3a57B22x6evXQMaQZ3T1vEi5+urfe4p35wOGff/W7U11n8u1PY99oXo07/6MWHcv69kUrbT8YP4tbJC6I+duLJ+3PX1EVs3d30bn+zrz+BB95eym1RXP/O8w9iwvCeUZ+7KeVOpXP8GMgH2gcVqz8Cm9x9kplNBDq5e70PdKIte8p/32rz85P2r6iMikhiqq/sibpiVf3JTl1pG1Kx2rSziOdVsRJpsZp6g2Nm3wUuBXYA84hUsL4dTcXKzC4BLgHo16/fwcuWaTKcVLGjsISj/vA6m3c1rALwyPcO5fLHPuK+iw7hjDvfbqboYmtw93Z8vm57/QlbuGlXj6V/lzZRpY1BudOHSGv4TcCPg4rV58BYd19jZj2Bqe4+uL5zxapiBbDk96dUtDCKSOKpr+xpyBirK4DPmh7Sl1R0iIi73+fuB7n70cAm4AtgXXBjQ/C6vpZj73H3fHfPz83NjV/QEnNzVm1lzqqte20vLCnlllc+Z3dR1bFTP39qdoMrVQDf+Mf7bNxZ1GIqVUBKVKqAiok64uQO4GdA5YFN3d19DUDw2i2eAQH86109HBJpyaKqWAVPdiYA/4h1ABpiJZLazKxb8NoPOAt4DHgeuDBIciGR7oKSpIpKyjj1L29x6l/eoqS0jJfnrMHdWbFpF2Mmvc5f31jIAb96mbyJL/DK3EhXu9rGIEnLVVwanxsCMzsVWO/ue8/GEf05LjGzmWY2s6CgIGax/fr5uTE7l4jEX7STV9xB5MlOu1heXM3dIgI8bWZdgGLgUnffbGaTgCeCboLLgXNCjVCa1aBfvFTx/g8vz+feN5dw+biB/HnKF3ul/f5Djb4XFik3BviKmZ0C5ADtzexhgpbySl0Ba2wph0hrOXAPRLoCxiNoEUl89bZYRftkp7FPb1wTroukNHc/yt2HuPsId58SbNvo7uPcfWDwuinsOCU+7n0zMs14TZUqSW4/PWFQXK7j7te4ex93zwO+Brzu7t9ELeUi0kTRdAUsf7KzFHgcOC54slNFY8Y6GOoKKCKSSt5dtJFJL82v+Dxrecxms04pj118WLOc92cn1TtXQ7M466DefOfIfUK5diWTgPFm9gUwPvgsIhK1ersCuvs1wDUAZjYW+GnwZKfJ1BNQRCT57Skupbi0jE9WbOWb90WmAb972iIA+nZuFWZo9frukfswa/nmirWronHV8YN4d/EG3lscXUPrz0/anzSD31eqcNbkjvNG8uYXG3h61kqG9enAlJ8cw7hbIxM+tM3OYEdhCbefN4Kr/v1Jred46LujueC+GTXuWzppAgAnDOnB1M/X06dTa7btKebsg/qQlmbMXrmFyx79iFVbdlNaFnkqOuO6cYy+aUqV84zo04HR+3Tm3jeXMKh7Wxas21Flf/ucDP5+QT6XP/4RxwzK5SsjenH0oHAmn3H3qUTWycPdNwLjmutab/x0LMfeMrW5Ti8iCSDUBYJBLVYiIsnswXeW1jkgf8Wm3XGMJmLs4Fymfl5/l/Ufjx/E5eMGcumjs/ho+RYO7NWeU4f34g8vz+fms4dz9VOz6dUhhyk/GctPn/qEYwbmcu4hkYVur2Agc1ZtZXCPdgy87iVuPP1A/vjK52zfU8K1p+zPtw7PY/9fvgzAIXmdyM/rzL65bfnXu0s58cAenDa8Fzf8by7/mbUKgNd+fAwDurXl9JG9uPns4aSlGW1z2/LoxYdyUL9O5GSmV8R95qg+vDxnLT94uGoP/u8fvS9j9uvKs5eOYVdRCZc/9jEbdhRyzwUHc8KBPSrSDejWlgHd2u71/RjepyPTrh4LwO2TF/Dn1xfSqXUWj37vUK7/71yuOn4QJw/7ci2q6yYMAWDh+h3cMz2yxtct5wznxAN7YGZ8cN3x0fy4kkb39tlVPg/v04HZK/eeCXPzziI6VVpvTURajlAXCP7egzNZvWU3L15xVMxjEJH4iMVCnbGgBYITUzRr9zTV784cxvmH9uOqf3/MMx+tqjf9mz87lqP++AYA5+b3YeXm3byzaCPtcjI4JK8zC9fv4PdnDWPMgK4ArN+2h9smL+CG0w8kKz2NPcVltMpKZ3dRKa2y0uu6VBVbdxezq6iEnh2+bKWr6xw7C0t4ac5avnpQ70ZN9vTEzBXc+urnrNtWyJmjenP7eSOr7N+0s4gNOwoZ1D2m81LFRaKUOxB92VNYUsrgX0Qq0wf168gt54zguFv3nmL+hcuP5MBeHWIep4g0XX1lT/gtVmEHICIiMffwe8v4xbNzmu38T/7gcNIMvnrXuxyxXxcAbj9vZEXF6s9fH8Xlj31EmkGZw30X5vPdB2dyzwUH07dza2ZcO4422Rm0yY78Gywtc9Ks5tlqu7XPYdJXh1d8Lq8INaRSBdChVSYdWmVW2VbXOdpkZ3D2wX0adI3Kzs3vy9jBuZx/7/v8ePzeE0N0bpNFZ7WMxE1G2pfD2k8b0Wuv34VyJXGadl5EYi/UipUZNEeLmYiIxM+igh24O306teaLdTvYsKMwZpWqdkHF59ZzR3DJQx/y7TF5TBjWk/y8zsCX44Kq+8qIXnxlRC+KSspwnOyM9Cppu7XPqZI+PS05B/12a5fDaz8+JuwwBKj8K3bREXmYGW9PPI4xk16nZ4cc1mzdA8AzH61iRN+O4QQpIk0SbsUqzIuLiEhMjKuhO1NjDOnZnnlrtlV8PnJAVx7+3qEVn2f+4ni6tMmqs1vcnecfxOAeX3Zty8qIZvJbkeZX+fe2/H3vjq347DcnkZZGRTfBwpLSUOITkaYLvSugiIi0TBt3FLJ6y56Yne8/PzyCRQU7eGfhRr531D57VaC6ts2u5cgvTRjes940ImEZ3qcD3zysf5Vt1buDHrpPl3iGJCIxFHpXQBERaXlmLNnEdx/8gO17Spp0nqd+cDgvfLqG+99eSk5mOgf26qCB+5K0nr/syFr33f3Ng/jBw7PoXq2bqoi0HKG3WGmIlYhIy3Pu399t8jkmDI+MlTq4fyd+deqQGEQl0nJ1ah2ZSKRMN0YiLVbIY6wM17yAIiIJbdnGnfTt1Jq0YPR9WVnDy+3/XnYkw/p0YMWmXXRuk0V6mpGZHhn/1JipxEWSTfkEKqWN+PsSkcSgroAiIlKr8nWorj5xMJceOwCAv09f3KBzzL7+BNrnRKaW7tu5dWwDFEkS5Q8uStViJdJiqSugiIjU65H3ltGhVSYDurXlDy/Pj/q4//3oyIpKlYjULt0a3yIsIolBLVYiIlKv1Vv3RL021YLfnszbizYwuHs7enVs1cyRiSQHdQUUaflCX+BDxYeISGJqzJPzU4f3JCsjjWMHd1OlSqQB0oKnze8u3hhyJCLSWKFWrExLBIuIJKzisrIGH9O7kypTIk1x/9tLww5BRBop/BYrDbISEUlIxaXRlc9XjBvIFeMGAnDa8F7NGZJI0koL/Y5MRJoq3MkrTF0BRUQSVXFJdC1WV40fVOVVRBouTQPPRVq8kLsCiohIovp83fawQxBpFmaWY2YzzOwTM5trZjcE2683s1Vm9nHwdUrcYorXhUSk2YQ+3bqarEREwvf2wg184x/v85Pxg7h18oKwwxFpboXAce6+w8wygbfM7KVg3+3ufku8A1KDlUjLF/J06ypFREQSwRWPfwRQY6UqJzONPcV7dws8/oDudGilNaqk5fHIAO8dwcfM4CvUR726JxJp+UIfKqkGKxGR8KzZupu8iS+wYUdRrWl+e8awivfvXTMOgLd+fiz/uDCfW88d0ewxijQHM0s3s4+B9cBkd38/2HWZmc02s3+aWad4xaMxViItX+hjrDQroIhIeN6LYs2c0rIyPvzF8bz246Pp0SGHpZMm0KdT6zhEJ9J83L3U3UcCfYDRZjYUuAvYDxgJrAFurelYM7vEzGaa2cyCgoKYxJOmepVIixduxUqFiIhIqN5csKHeNO7QpW02A7q1i0NEIvHl7luAqcBJ7r4uqHCVAfcCo2s55h53z3f3/Nzc3JjEobU9RVo+dQUUEUlBC9Zt560vNvCfj1bVm/aMUb3jEJFI/JhZrpl1DN63Ao4H5ptZz0rJzgTmxCumru2yAMhQ05VIixXu5BVhXlxEJIWdcPv0qNPmZKY3YyQioegJPGhm6UQeMj/h7v8zs4fMbCSR575Lge/HK6DWWRm0y8ng7IP7xOuSIhJjoU+3riFWIiIiEk/uPhsYVcP2C0IIp0Jk7HmYEYhIU4Q8xkptViIi8bZlV+0zAIpIeNLSTJN6ibRg4bdYaZSViEhcDPnVy3Rrl81B/eueQfqTX51AdmYa+//y5ThFJiIQabEq022RSIuVANOthxmBiEjq2FVUytKNu/jPrLonrGjfKkPjqkRCsHlXMQ+9tyzsMESkkcKdFVA9AUVEEo66aYuIiDRc+NOtq8VKRKTZadyGiIhI8wq5K6CeioqINKelG3aSN/EFpn9R+0LAb/38WJZOmkBmuspkERGRxgp98goREWk+s5ZvBuDCf86oNU2fTq0BePnKo5m1bHNc4hIREUk24S4QbOqeIiLSXO54bQF3vPZF1On3y23LfrltmzEiERGR5BVuxSrMi4uIJLn6KlWf/OoELPSRtiIiIskh9K6Aaq8SEQlHh9aZYYcgIiKSNELvCigiIrHl7tz75uI607TLDv25moiISFIJ/T+rhliJpDYzuwr4HpEG7E+BbwOtgX8DecBS4Fx316wKUZi9cguzV27ldy/OrzXNEft14ScnDK73XH06taJ/l9axDE9ERCRp1VuxMrMcYDqQHaR/yt1/HYuLG4arM6BIyjKz3sDlwBB3321mTwBfA4YAU9x9kplNBCYCPw8x1BbjK399u9Z9xx/QjT+ePYLObbKiOtdbPz8uVmGJiIgkvWiGLRcCx7n7CGAkcJKZHRaLi6sroIgQeWDTyswyiLRUrQZOBx4M9j8InBFOaC1LXbOsDu7ejnu/lR91pUpEREQapt6KlUfsCD5mBl8xa2ZSV0CR1OXuq4BbgOXAGmCru78KdHf3NUGaNUC38KJsOT5dtbXWfa9cdTSmp1kiIiLNJqoxVmaWDnwIDADudPf3Y3Fx/Y8XSW1m1olI69Q+wBbgSTP7ZgOOvwS4BKBfv37NEWKLsH77HkbfNKXW/UcN7BrHaEQSX23DHMysMxrfKSKNFNUKJu5e6u4jgT7AaDMbWj2NmV1iZjPNbGZBQUHUAajBSiSlHQ8scfcCdy8G/gMcAawzs54Awev6mg5293vcPd/d83Nzc+MWdKI55+5369z/0HcPjVMkIi1GbcMcJhIZ3zkQmBJ8FhGJSoOWhnT3LcBU4KQa9jXiBkdNViIpbjlwmJm1tkg/tXHAZ8DzwIVBmguB50KKr0VYtnFX2CGItCh1DHPQ+E4RabR6K1ZmlmtmHYP3rYg8Ya59Ht8G0hgrkdQVdCt+CphFZKr1NOAeYBIw3sy+AMYHn6WaOau2kjfxhbDDEGmRzCzdzD4m0iI+OSiPNL5TRBotmjFWPYEHg3FWacAT7v6/WFw8MsZKNSuRVBYs31B9CYdCIq1XUodT//JWnft/fdoQDsnrHKdoRFoWdy8FRgYPj5+paZhDbTS+U0RqUm/Fyt1nA6Oa4+LqCCgi0jgfLN1Ub5pvj9knDpGItGzuvsXMphIZ5rDOzHq6+5r6xncSaV0nPz9fT4hFBGjgGKvmoK6AIiINU1xaVu+EFX06tYpTNCItTx3DHDS+U0QaLarp1puLplsXEWm41+atqzfNYft2iUMkIi1WjcMczOxd4Akz+y6RyXXOCTNIEWlZQq1YgUZYiYg0xKKCHfzfI7PqTdc+JzMO0Yi0TLUNc3D3jWh8p4g0UqhdAQ3D1RdQRCRq426dVm+aob3b85MTBsUhGhERESkXbsVKXQFFRKJW34Oo288bAcD9F42mTXboHRJERERSSuj/edVeJSISnbunLa5z/5mj+nDmqD5xikZEREQqC3fyijAvLiLSQrg71z07h0ffX17j/qd+cDgDu7eLc1Qi0lzKypy0NN0libQ04bdYqclKRKRWL89Zw7KNu2qtVC35/SmY+lWLJJXlm3aR17VN2GGISAOFPN26bgZEROryg4frngFQ5aiIiEhiSIAFgtVkJSJSrmB7IVt3FQPwx5fn15n2k1+dEI+QRCTOdGck0jKF3xUw7ABERBLIITe9RlZ6Gu9ecxx/m7qozrQdWmutKpFkpIfOIi2TplsXEUkwRaVlHPzb1+pMM//Gk+IUjYjEW2FJWdghiEgjhN4VUE1WIiLw8Yot7CoqiTp9dkb4xbeINI/bJy8IOwQRaYSQp1tXk5WIyMYdhZxx59tRpX3/2nGs2bpHk1aIJLF12wvDDkFEGkFjrEREQrZld3HUabu3z6F7+5xmjEZEwlZWprsjkZYo9DFWGqApIqls3bY9jLt1WlRpu7fPbuZoRCQRaG1gkZYp3IpVmBcXEUkAk16qe0r1yob17th8gYhI6I4dnAuAGqxEWqbQRz+r7BCRVPbMR6vqTbPwppP5zph9+N1ZQ+MQkYiEJS0YO1mm3jwiLVLoXQFFRFLNX6Z8wacrt0aV9q/njyIjPY1fnTaEbu00tkokmaUHfQBL1WQl0iKF32KlskNEUsytkxdw2l/fqjfdD8fux6nDe8UhIpHUYmZ9zewNM/vMzOaa2RXB9uvNbJWZfRx8nRLPuI4eFOkK2LdzayY+PZs9xaXxvLyINFHILVaGqzOgiKSQhkzY883D+jdjJCIprQT4ibsfABwGXGpmQ4J9t7v7yODrxXgGdfLQHgBMnreOxz9YwXMf199VWEQShyavEBGJgz3FpTw5c0WVQelPzFxR5zHqLi3SPNx9jbvPCt5vBz4Deocb1ZddAcut2LQ7pEhEpDHUFVBEJA4mvTSfq5+azY3/m1ex7WdPza4x7e/PGkavDjl0aaPp1UWam5nlAaOA94NNl5nZbDP7p5l1inMste5bsG47Wxuw5p2IxF+4FSs9jRWRFLFma+TJ8wPvLK037ddH9+Oda8aRlRH6sy+RpGZmbYGngSvdfRtwF7AfMBJYA9xay3GXmNlMM5tZUFAQs3iqt1hVdsLt0zn37ndjdi0Rib3Q/2urwUpEUsErc9dFlW54nw7NHImIAJhZJpFK1SPu/h8Ad1/n7qXuXgbcC4yu6Vh3v8fd8909Pzc3N2YxVa9XFZZUnbzi83XbY3YtEYm9kMdYqclKRKSyW88ZEXYIIknPIn3u7gM+c/fbKm3vWSnZmcCceMaVnZFe5fOLn66N5+VFpIkywg5ATVYikuyinQnwwsP7M7B7u2aORkSAMcAFwKdm9nGw7Vrg62Y2ksjdyVLg+/EMqnpXwFVbNHmFSEsSasXKDE23LiJJ77mPV0eV7sfjBzdzJCIC4O5vUfNI77hOry4iyUXTrYuINKMVm3Yx6aX5daY56cAe3PWNg+jQOjNOUYlIovrOmH3CDkFEGin0roCabl1EktHKzbvo06k1R/3xjXrTjt6nMycP61lvOhFJfoN7tK3yOW/iCyFFIiINFXpXQBGRZDNjySbO/fu7HDmga1TpNa26iJQrKav7ifOMJZsYvU/nOEUjIg0R+n9zNViJSLJZEEyJ/NbCDVGlV8VKRMqV1lOx+t/s6MZsikj8hT7derSzZYmItBQNbY3PVsVKRAL1Vax02ySSuMKtWKkroIgkmR2FJewsLGnQMVnpqliJSER9FauH3lvGJyu2xCcYEWmQ8CevCDsAEZEYKCop44Olm/jGP95v8LHqCigi5QZ0a1tvmtPvfJulkybEIRoRaYh6/5ubWV8ze8PMPjOzuWZ2RawurgYrEUkWf3h5fqMqVQBpaSoNRSRi7OBuTPnJMWGHISKNEM1j0hLgJ+5+AHAYcKmZDYlVAOorLCLJYFHBjgalrzyrl6pVIlLZfrn1t1oVbC+MQyQi0hD1VqzcfY27zwrebwc+A3rH5OoaZCUiLZy7U1RSxp7i0gYdN6Rn+4r3GWnqCigiVT176Zg69ze0zBGR5teg/+ZmlgeMAhrX30VEpBIzG2xmH1f62mZmV5pZZzObbGZfBK+dwo61Nre8+jmDfvES7y3eFFX67x25D3ecN5Irxg2s2JauroAiUs3Ivh1ZOmkC3ztynxr3b9tTHOeIRKQ+UVeszKwt8DRwpbtvq2H/JWY208xmFhQURHfO4FVTroukJnf/3N1HuvtI4GBgF/AMMBGY4u4DgSnB54RTUlrGnW8satAxvzh1CGeM6k2nNlnk94/UFzPSVbESkZr94tSaR1/8+rm5cY5EROoTVcXKzDKJVKoecff/1JTG3e9x93x3z8/NzY3q4uoJKCKVjAMWufsy4HTgwWD7g8AZYQVVm+LSMv42tWGVqmlXj63yuSSYVjlNhaGI1OGCw/rvtW3l5t0NXtpBRJpXNLMCGnAf8Jm739YcQajBSkSArwGPBe+7u/saiIzzBLqFFlUtBl73ErdNXtCgY/p3aVPlc/l6NRnqCigidejaNnuvbWu37eHAX78SQjQiUptoWqzGABcAx1UaB3FKLC5umgtLRAAzywK+AjzZwOMa3AU53i45et9a9500tAcAPTvmxCscEWmBjhlce0+g2xv4gEdEmk80swK+5e7m7sPLx0K4+4uxDEINViIp72RglruvCz6vM7OeAMHr+poOakwX5Kbauqs4qkHjN589nKWTJnDNyfvXmuaHY/fj0+tPoFs7VaxE4qm2NToTdeKc8oksavKnKV9w3TOfsnD99jhHJSLVhTrHb/mwAk1eIZLyvs6X3QABngcuDN5fCDwX94hqMeI3rzL8+lfrTXdOfl8ALCjo2mSl75XGzGiXkxnbAEUkGrWt0ZnQE+c8f1nNU7A/8v5yLnnowyrbCktKWbZxZzzCEpFAuBWrMC8uIgnBzFoD44HKE+NMAsab2RfBvklhxNZYHVtXrSzdeMZQnv/RkSFFIyLV1bFGZ0JPnDOsdwcuPqrm6dcXF+wkb+ILFZ+vfnI2x9w8ld1FWu9KJF4ywg4A1BVQJJW5+y6gS7VtG4nMEpgQJr00H4CJdXTrK/fHrw7n1BE9q2yraUYvEUkM1dborDJxjpkl1MQ5ZsZ1E4Zw75tLak2zbONO+ndpw5TPIj2rS8rKgL1bzEUk9hKiK6CISCK7e9oi7p62iF1FdU9t3C47g3MP6UvrrIR4ZiUi9ahvjc46jkvYiXOOuXkqT8xcQXEw6+h/P1lDWZkeYYvEQ6gVq3IaYiUiLcENz8+rdd/s60/g0xtOjGM0ItIUtazRmbAT51Q2/8aTePA7o2vd/7OnZlNUUgbAtc98ylOzVlbZv3brnqgm4RGRhgm5xSrSZOXqDCgiLcC/Z66ocfsJQ7rTXpNQiLQYdazRmbAT51SWk5nOMYOir9Bt3FFU5fNhv5/C+NumxToskZSXEC1WIiIt2VEDu4Ydgog0TG1rdLaoiXMuOiIvqnQ3vzKfT1ZsAb6ciXndtsJmikokdSVExUpdAUWkJfumJqcQaVFqW6PT3Te6+zh3Hxi8bgo71rr88tQhUaUrczjrrncAOPR3U5ozJJGUpskrRESayFSYiUgI0tOiL3tKgwks1m9XS5VIc0mIFisRkUS1ZuvuOve/fOVRcYpEREREElmocwKblggWkQRXvoZVTZZOmhDHSERE9va3bxxE5zZZfO2e9+pNu3pL3Q+KRKRpEqLFSmOsRCRRZaQlRDEpIlKjU4b15LB9u/Dmz46tN+0Rk16PQ0QiqSshxlhpunURaWnyurQOOwQRkQp9O7fmxtMPDDsMkZQWbsUqzIuLiNTjZ099wtPVFtYs16lNVpyjERGp2wWH5/GdMfuEHYZIykqIPi7qCigiiWbbnmKemFlzpQrgjJG94xiNiEh0OrfRYuUiYUmIroAiIolmfT2LZ559cJ84RSIiEr0TD+wRdggiKSsxWqzCDkBEpJrnP1ld5fPjlxxW5XOangyJSAIa2L0dSydN4KYzh9ab1tVlSCSmQh5jFbkx0R+2iCSaj5ZvrvJ5RJ+OPPmDwys+q14lIons9JG9yW2XXWeatxZuiFM0IqlBXQFFRGrw5hdVbzgy0o1D8jpzUtDNJj1NBZiIJK622Rm8f824OtNccN8M5qzaGqeIRJKfugKKiEQhI6hI3fG1kbz5s2PJTE+I4lNEpFZpUTwA+upd78QhEpHUoDsDEZEoWNDEnpOZTt/OWsNKRJJDYUlZ2CGIJI2EqFhpiJWIiIiIiLRkIY+x0hgFERERkTBt2lkUdggiSSEhWqw0yEpEEs0R+3WpeN+nU6sQIxERabyLj9qHCcN6Mu83J1bMbJqVUfX276AbJ4cRmkjSyQjz4uXtVa6alYgkmHcWbQTgmR8eQT+NqRJJKmb2T+BUYL27Dw22XQ9cDBQEya519xfDiTB2rpswpOL9IXmdefWqo3GHE++YHmJUIslJ062LiFRTeW29Uf060aVt3WvBiEiL8wBwUg3bb3f3kcFXi69U1WRQ93Z0q2F9q2Nvmcorc9eGEJFI8kiIroCavEJEEsnu4tKwQxCRZuTu04FNYccRlpzM9L22Ldmwk+8/9GEI0Ygkj3BbrMK8uIhILRat3xl2CCISjsvMbLaZ/dPMOoUdTHPJyaz99u/1+eviGIlIckmMFquwAxARqWTN1t1hhyAi8XcXsB8wElgD3FpbQjO7xMxmmtnMgoKC2pIlLDPj5KE9atz3nQdmkjfxBe54bUGcoxJp+RJiunVXX0ARSSAqkURSj7uvc/dSdy8D7gVG15H2HnfPd/f83Nzc+AUZQ3d98+A699/x2hfcPW1RnKIRSQ6avEJERERSnpn1rPTxTGBOWLEkikkvzWePxpyKRE1dAUVEqlEjukhyM7PHgHeBwWa20sy+C/zRzD41s9nAscBVoQYZB8cf0L3eNPv/8mX1LBKJUkKsYyUikkh0EyGS3Nz96zVsvi/ugYTsb984iB2FJUyet5afP/1prekeeGcp3x6zTxwjE2mZEqPFSvcwIpJAylQmiUgKyMpIo3ObLM47pB9LJ01g4sn715juhv/Oi3NkIi1TuBWr8skr1BlQRBJImZ72iEgK+sEx+3HWqN5hhyHSYmkdKxGRalStEpFUNbJfxxq35018gV89l/LzeYjUqd6KVbBI3noza76/Jt3FiEgC0RgrEUlVFxzWv9Z9/3p3WRwjEWl5ommxegA4qTkurunWRSQRqV4lIqnKzPjPD4+odf+TM1fEMRqRlqXeipW7Twc2NWcQuocRkUSiMVYiksoO6teJl644ii9uOnmvfVc/NZs731gYQlQiiS/kMVZqshKRxKNZAUUk1R3Qsz2Z6WkcsV+Xvfbd/Mrn6jItUoOYVazM7BIzm2lmMwsKChp0rP42RSSR6IZBRCTiqIG5NW7XFOwie4tZxcrd73H3fHfPz82t+Y+wuvIxVppuXUQSiepVIiIRFx+1D2MG7N1q9cA7S+MfjEiC03TrIhIqM+toZk+Z2Xwz+8zMDjezzmY22cy+CF47xTMmjbESEYnISE/jke8dRl6X1nvty5v4Alt3FYcQlUhiima69ceAd4HBZrbSzL4b6yB0DyOS0v4EvOzu+wMjgM+AicAUdx8ITAk+x42KJBGRqgZ2b1fj9junaiILkXIZ9SVw968318U13bpIajOz9sDRwEUA7l4EFJnZ6cDYINmDwFTg5/GKSy1WIiJV1Tb29J7pi7ln+mLaZWfw668cyNkH94lzZCKJI9SugOV0CyOSsvYFCoD7zewjM/uHmbUBurv7GoDgtVs8g1K9SkSkqnPy+9a5f3thCT998pM4RSOSmBJiunXNwCWSsjKAg4C73H0UsJMGdPtrymykdVGZJCJS1YkH9mDppAkcNbBrnelKSsviFJFI4gm3xUpdAUVS3Upgpbu/H3x+ikhFa52Z9QQIXtfXdHBjZiONhtaxEhGp2X0XHlLn/l88OydOkYgknsToCqibGJGU5O5rgRVmNjjYNA6YBzwPXBhsuxB4Lp5xaYyViEjNsjLS6NOpVa37H/9gRRyjEUksmm5dRML2I+ARM5sNjAR+B0wCxpvZF8D44HPc/OvdZfG8nIjEmZn908zWm9mcSttCXeahJXnxiqPCDkEkISVEi5WIpC53/zjozjfc3c9w983uvtHdx7n7wOB1UzxjWrJhZzwvJyLx9wBwUrVtoS7z0JK0z8lk6aQJte6/8X/zKFOfaklB4bZYab51ERERiTN3nw5Uf2BzOpHlHQhez4hnTC3R9acNqXH7fW8tYfmmXXGORiR8CdFipeEMIiIiErJQl3loiS4asw+Trzq6xn1Fmh1QUlBCjLFyrWQlIiIiLURzLfXQEg3s3o6+nfeezGL5xl3qDigpJ+SugGFeXUSkZmcd1BuAzHQVUiIpJKplHqD5lnpoqV658mg+/tX4Ktu+96+Z7Hvti3y8Yks4QYmEQF0BRUSq6dgqC4CcjPSQIxGROAp1mYeWrHVWBh1bZ9W474w732bh+u1xjkgkHGqxEhGppmIdK5VRIknJzB4D3gUGm9lKM/suIS/zkAx+PH5QjduPv2060xakdpdJSQ2J0WIVdgAiIjVI09MfkaTk7l93957ununufdz9vrCXeUgGx+1f+3wfF/5zBh5FF6WF63cw4oZXWb1ldyxDE4mLkCeviNy0RPOHJiISL6XBgOs01atERKI2tHcHpl09ttb989Zsq/ccj76/nK27i3nx0zUxjEwkPtQVUESkmvKZStViJSLSMP27tOGBbx9S474Jf36L3UWlcY5IJH7UFVBEpJqKIVaqV4mINNjYwd0Y1L1tjften1/rZIuAyl1p2TLCDkBEJNF0Cma3Om1Er5AjERFpmV696hg+W7ONk//0ZpXtlz46i/Xbh9CxdSZnjuoTUnQizSMhKlYaYiUiiaR9q0jRWNsMVyIiUr8DeravcfsN/50HwOkjepNWy2BW3RtKSxTyGKvyPyb99YhI4gjmriBds1eIiDTJ3BtOrHVfUWnZXttU6kpLFvKsgCIiiaf8SakmrxARaZo22RmMH9K9xn01VaxEWrLEmLxCDVYikkDKFwhWvUpEpOnu/VY+s68/Ya/t7y/exNqte7hn+iIWrNseQmQisRXqGCvdtIhIIiopDSpWalcXEYmJ9jmZe227+F8zK97/7sX5LJ00oeLzKi0QLC1QYrRYhR2AiEglt7+2ANACwSIi8bZpZxEAD7yzNNxARBoh5DFWumsRkcSlMVYiIrEzsFtkbasrjx9Ya5oyjQ+RFiwxWqz0N5Sy3J2ysnB+ATbsKKS42sDZgu2FbN5ZRElpWZV97s6nK7fudQ5vwC/vh8s2s21Pca37dxeVUlRS+0Bed2f99j0NimH1lt1s3V3zNcuPc/d6r52qVK8SEYmdFy4/ivk3nsSVx9e8lMXMpZuYuWxznKMSiZ2EGGPl6gzYIu0sLKG4tIz2OZmUubN1dzFLN+6iQ6tMWmWl0zY7g1fnruXqp2Zz0oE96NEhh+7tc/jDy/MbfK12ORls31PSDLmQmiz47clkZSTEc5dQmWpWIiIxU9//lbPvfjdOkYg0j3ArVmFeXGq0p7iUgu2F9OnUisKSMrbtKebe6Yt5ee5aVm3eTV7XNvTq0Iq3Fm5o0Hlfnru2SXGpUhVf6oohIiIi0jChVqzK6R4uPty94gl8aZmzbONOfvzEJxRsL4x69p3FBTtZXLCzOcOUkP1iwgHkZKaHHYaIiCSxzm2yKiaqEEkWCdEVUGKnrMx5b8lGDt+3CwA7i0qZuXQTc1dv4+ZXPm/26z/4ndH84tlP6dwmm8/XbqNLm2z+/PWRvLd4E+1bZXLWqN5s2FHI7JVbOXb/bsxesYWR/TryzsKNlJQ5Rw/qSqvMdHYWlZKdkUZxaRnvLd7I0QNz2VVcys7CErq2zcaAtxdtZGTfjmSkGUUlZXRqk1URR8H2QlZs3sVB/TpV+d4sLNjBoO7tGpSn3UWlZKYbGenN1zWurMwxa3jXs+17immVmd6ssYmIpBIzWwpsB0qBEnfPDzei5PTGT8eyu6iUw34/JexQRGJGLVYtyPy122iXk0nvjq0qthWVlPH52u2c9te3muWa5xzchyc/XMmrVx1N17bZZGeksWzjLtq3yqBPp9Y1HvPmz47ba9vB/TtXvG+TnUH/Lm0AOGJAVwCOr7Yqe9vsyK9mZnoax+0f2dc+Pa3KOhjHDMqtdM6q18ttl01uu6ob09KswZUqgFZZzd96k9bIeb3b1bAuiIiINNmx7t6wPu/SIB1aZdKhVSY3nTmUfp1bc8F9M/ZKs3lnEaNunMw9FxzMgb07sG7bnioPTEUSTcgVq8jNpCavqNuuohKG/OqVis/3f/sQvn3/BzE597OXjqFT60yKS52s9DT6ddm7snTzOSOqfB7Sq31Mri0iIiKp7RuH9gfgt2cM5frn5/LC5Udx4h3TARh142QALnnoQ9IMypwqiwiLJBp1BUww7s7zn6zmlGE9WbJhJys37+KWVxZUSdOYStW3x+Rx6bEDeG/xRsYO7lbRIiQieztivy57TcUvIinDgVfNzIG/u/s9YQeUCr55WH++eVj/WveHtDKLSIMkxN11KncF3FVUwhfrdpCZnsbtry1g8rx1AFzx+McNPtdZo3qzZXcx15y8P7OWb+bc/L57jdk5dXivWIQtktRKypz0RnbPFJEWb4y7rzazbsBkM5vv7tMrJzCzS4BLAPr16xdGjEntoiPyeOCdpTXu27qrmA6t1Q1eEpOmW4+j1+atY79ubbnvrcU8/N7yJp3ruUvHkNelDa/OW8tJQ3vsNdZmYCPGEolIRGmZk6WZEUVSkruvDl7Xm9kzwGhgerU09wD3AOTn56fw4+Hm0al1Vq37RvzmVW46c2hFF0KRRJIQLVbJaMWmXcxdvZWThvYE4NDfvca6bYWNPt8JQ7rz9wsO3qsF6pz8vk2KU0T2phYrkdRkZm2ANHffHrw/AfhNyGGlnJWbd9W5/7pn5nBefl/NiCsJJ6qKlZmdBPwJSAf+4e6TYnHxhk4tneg27yzixTlrmLVsC0/PWtno8wzr3YHNu4rYtruY2defGMMIRSQapWVlZKhiJZKKugPPBPcnGcCj7v5yuCGlnmWb6q5YAdzy6gLunraI/152JMP6dIhDVCL1q7diZWbpwJ3AeGAl8IGZPe/u82IVREsbY/X52u307tSKXYUljP5d09ZfOGtUb1pnp/Oj4waSkWZ0aZtd/0Ei0qzmrNrGHLaFHYaIxJm7LwZG1JtQmtWMJZsq3memG8Wle98o3j1tEQA/f3o289Zs4+Urj2L/Hpq1WMIVTYvVaGBhUNhgZo8DpwNNrliVPw9O1OnWy5uiu7XLISsj0tzs7hXTgDbWcft3Y9+ubbj2lAMavX6RiDTeOws38NSHK7ntvJFhhyIiItX8+5LDOO+e9wC48vhB3PzK5xzYqz1zV+/9wGvemsi2V+asU8VKQhdNxao3sKLS55XAobG4eHlPwCUbdjK8T8dYnLJB1m/fwz/eXMLYwbkMyG3L9sISlm3cyRvzCzhjVG++etc7MbnOqH4d+ev5B1VZ2FdEwnP+P94H4JZzRlQ83Pho+WYu/OcM/vejo8IMTUQk5R26b5eK9/93zH6ceGAPcttlM2v55lqXnHnh09UM6NaWrbuLOf9QzdQo4YimYlVTk8peTUxNmXr0isc/5vSRvRt0TCyMvinSje+e6Yv32vfQe8sadc7xQ7qzp7iUv19wMK2zNDeISCLb99oXef6yMQzv05E731jEtj0lHH3zG2GHJSKS8l778TFs31NMWpoxoFtbAI4d3K3W9AvW7eDSR2cBqGIloYnmzn8lUHnquT7A6uqJGjP1aLznrijYXsjYm99gZ1Eplx07IGbn/e9lR9KxdSZ9O7eO2TlFJD6+8te3WTppghYsFxFJIOWVqer+9LWR9a71mTfxBab+dCx5Xds0Q2QitYumYvUBMNDM9gFWAV8Dzo91IO4e81kCt+4upkOrTLbsKuLh95Zxy6sLKvb99Y2FjT7vtafsz8VH7Zt0sxqKiIiIJLLTR/auqFh1bJ3Jll3FNaYbe8tUAJ69dAwdW2WqkiVxUW/Fyt1LzOwy4BUi063/093nxuLiVqmX4UX3f8DfLziYnBgsynnnGwtZuXkXj81YUX/iKM2+/gTa52ilb5Fk9NzHq5g8b13YYYiISBRm/XI8RSVldGqTyYpNuzn+tmm1pj3jzrcr3j/9f4czqm8npi0oYOzgXD0gl5iLahCQu78IvBjzq1f6fZ62oIDX56/nlGE9efT95fTskENuu2xO/ctbZKQZk398DPtUetpQsL2QQ256jRvPGMoFh/Xnumc+5ZH3lzc5pOyMNPp2bs2Pxw9iYLe2PPXhSr51RJ4qVSJJrL5uJSIikjg6t8mqeD+gW1tu+MqBFJaU8sDbS1m9dU+tx331rne56cyhXPfMHG47dwRnHdQnHuFKCkmo2RX+8PJ8ThnWk2uf+bTK9pIy59hbpjJmQBfeXriRrm2zyEiLTH/+y2fn8Mtn5zT52uUVtOquOeWAJp9bRGpnZkuB7UApUOLu+WbWGfg3kAcsBc51982xuJ63tIXzRESkThcekQfAxUfty8L1Oxh/e+3L4vz19chQkJWbd8cjNEkxaWFevHoD7LKNu7jo/hm1pn974UYANuwoYu222p9IROP9a8dx/0WHVHyuqVIlInFzrLuPdPf84PNEYIq7DwSmBJ9joqRMFSsRkWRkZgzs3q7ONGuCFq3bJi9g1vLNbN9TzEX3z2DTzqJ4hChJLqFarACmfl4Q83PeePqB/PK5uTx/2RgGdmvH2m176N4+h+7tc7j1nBF0bZcd82uKSJOcDowN3j8ITAV+HosTF5WUxeI0IiKSoN67ZhzZGWl0aJXJvtfWPpLlrL99uV7pQTdOZumkCfEIT5JYqBWr5ho0eNTArhwzKJfSMuerB/eha9tsLjg8r2J/5bFaXz1Y/WtFQubAq2bmwN+DpRu6u/saAHdfY2a1L17SQMWl0VWsLh83MFaXFBGROOrRIafi/bOXjqkygUVd8ia+QOusdF68/Cie+WgVa7bu5o9nj2iuMCUJhVuxivH5/vS1kaEsNCwiTTLG3VcHlafJZjY/2gMbszD5nFXbokp38tAe0YYhIiIJamTfjiydNIFlG3dyzM1T602/q6i0Yqp2gCdmrgQia5Zu21NMjw457Jdb8xpbIgnXFbA24/bvxruLI2Osrjx+ID07tKJzmywO7t+J1+ev56B+nao8oRCRlsHdVwev683sGWA0sM7MegatVT2B9bUc2+CFyUujnLzigJ7to0onIiKJr3+XNhVd/YpKyvjvJ6v5yZOfRH38aX99q+L94O7teOh7o+nWLoeS0jKWbtxV64LGklpC7gpY9/6lkyZw6l/eZM6qbdx9wcFkptc818Ypw3o2Q3Qi0tzMrA2Q5u7bg/cnAL8BngcuBCYFr8/F6pqj8zrXm+bcfHURFhFJVlkZaXz14D4Vw0GWbthZpZWqPp+v287om6bUmSYjzejXpTWLC3Zy6bH7cecbizjxwO7cfM4I2mRlsGrzbkrKyliwbgcnHtgdgMKSMnYWlrB0404O7l/z/6qdhSVkZaRV3BOv2bqbbu1ySE/b+6Z67uqtbNtdwuH7dYk6b9I0oVasqj84nnPDiewpLuW1eetIC2pdD357NJ+t2V5rpUpEWrTuwDPBeMsM4FF3f9nMPgCeMLPvAsuBc2J1wVZZ9S9CfvWJ+8fqciIikuDyun7ZmvXnKV9w2+QFZGekUdiEyY5KypzFBTsBuPONRQC8Mncdr8x9tdHn/O6R+3DfW0sAuOzYAazespv/fLQKgEHd27JuWyHd22fz2zOGce7f36047n8/OpLP127niZkrOCe/L+OHdCcjzcjKSOOqf3/M947al5F9O7JpZxHz125jQLe2LCnYyTMfreKK4wfyjXvf51uH9+eU4T3JTk9ne2ExR/7hDQAuOiKPB95ZyrSrx/LaZ+s5ZlBXFq7fwYkH9mDFpt28v2QjvTu1YuOOIo4elEuHVpks37gLM+jePofP1mxj39w2ZKan8cLsNZw6oieXPjKL1z5bz5wbTuT9xRspLnXy8zqxZsseDujZjhWbd9OnUytWb9lN/y5t2FNcSsH2Qp78cCU/Om4AZe5kZ0T+12/dVczGnYVkpqdhBr07tqqY46GwpJSs9LSYzvlgzbGmS35+vs+cObPedJPnrePif32ZTrOxiLQ8ZvZhpWnSQxNtuQORAcp1UVkkktgSpdyBhpU90rKs37aHJRt2snV3MZc89GHY4UgzmfebE2mdFV1bU31lT6gtVgvWba94P6i7+qaKSHj279GO+Wu3c/FR+4QdioiEyMxOAv4EpAP/cPdJIYckIenWPodu7SPj9ys/cCstcx5+bxm/fn5uWKFJDOVk1N+TJVqhVqxWbt4FQHZGGi9cflSYoYhICvnkVyfwf498yDuLIhPiPPWDw+nfpQ0X3Pc+Fx6RF25wIhIaM0sH7gTGAyuBD8zseXefF25kkkjS04wLj8ir9f9FaZmzu7iUttmR2+yyMscsMobqyQ9XMmflVnp0yGHf3Da8s3AjvTq24rLjBrBtdzFPfbiSrIw0pi0o4PX5Vedtym2XTcH2worP3dplk9sum7mro5vtVmqWVsP4tMYKtWJ14+lDSTPj16cdqDFUIhI3HVpn8s+LDuGG/87lh2MH0LdzawBevvLokCMTkZCNBha6+2IAM3ucyILlqlhJ1NLTrKJSBV/euOdkpnPBYf2rpK28TFCnNllcfPS+AFx4RB6bdhYB0LlNVsxjdHc27CiiQ6tMytzJyUyvsq+otIw0s73uz/cUlzJr+WaO2K8rm3YW4cGxZrB80y7279GesqBimWZGdkZalYrLR8s3M6JPxyrb3J3pX2zg6IFdq4x32lNcyhvz13Nw/07ktssGal8Dt3xoU/n+sjJn/fZCWmens2zDLj5bu419u7ZhT3EZB/XvSHGpM2/1Ng7sHdsZgEOtWGWkp3HTmcPCDEFEUlROZjq/P2t42GGISGLpDayo9HklcGhIsUiKa44KVTkzq6is1LQvu5bucTmZ6RyxX9ca49u/R6SSkpZmtMmuuYoxql+nGq93zKDcGq91cpQzf1evcKWlWcUyTMP6dGBYnw57HdMcsyWqmUhEREQkoqbH4XvN8mVml5jZTDObWVBQEIewRKQlUMVKREREJGIl0LfS5z7A6uqJ3P0ed8939/zc3L2ftItIalLFSkRERCTiA2Cgme1jZlnA14gsWC4iUq9Qx1iJiIiIJAp3LzGzy4BXiEy3/k9315zaIhIVVaxEREREAu7+IvBi2HGISMujroAiIiIiIiJNpIqViIiIiIhIE6liJSIiIiIi0kRWvlJxTE9qVgAsizJ5V2BDzINILKmQR0iNfCqPe+vv7qHPN9zAcgf0s0wWymPyaEg+E6LcAd3z1CAV8gipkU/lcW91lj3NUrFqCDOb6e75oQbRzFIhj5Aa+VQek0cq5FN5TA6pkEdIjXwqj8kjFfKpPDacugKKiIiIiIg0kSpWIiIiIiIiTZQIFat7wg4gDlIhj5Aa+VQek0cq5FN5TA6pkEdIjXwqj8kjFfKpPDZQ6GOsREREREREWrpEaLESERERERFp0UKrWJnZSWb2uZktNLOJYcXRWGb2TzNbb2ZzKm3rbGaTzeyL4LVTpX3XBHn93MxOrLT9YDP7NNj3ZzOzeOelNmbW18zeMLPPzGyumV0RbE+afJpZjpnNMLNPgjzeEGxPmjyWM7N0M/vIzP4XfE66PEajJZc9KneSI58qd5Irj9FS2ZPYP0uVPcmRx3KhlT3uHvcvIB1YBOwLZAGfAEPCiKUJeTgaOAiYU2nbH4GJwfuJwB+C90OCPGYD+wR5Tw/2zQAOBwx4CTg57LxVyk9P4KDgfTtgQZCXpMlnEE/b4H0m8D5wWDLlsVJefww8CvwvGX9fo/wetOiyR+VOcuRT5U5y5THK74PKngT/WarsSY48VsprKGVPWC1Wo4GF7r7Y3YuAx4HTQ4qlUdx9OrCp2ubTgQeD9w8CZ1Ta/ri7F7r7EmAhMNrMegLt3f1dj/wE/1XpmNC5+xp3nxW83w58BvQmifLpETuCj5nBl5NEeQQwsz7ABOAflTYnVR6j1KLLHpU7yZFPlTvJk8cGUNmT4D9LlT3JkUcIt+wJq2LVG1hR6fPKYFtL193d10DkDxToFmyvLb+9g/fVtyccM8sDRhF5upFU+Qyaiz8G1gOT3T3p8gjcAfwMKKu0LdnyGI1kLHuS9ueocqdl5xGVO5Wp7GlBP0uVPS07j4RY9oRVsaqpj2IyT09YW35bxPfBzNoCTwNXuvu2upLWsC3h8+nupe4+EuhD5CnF0DqSt7g8mtmpwHp3/zDaQ2rYltB5bIBkyEO0WvTPUeVOFS0ujyp39pIs+YhGi/5ZquyposXlMeyyJ6yK1Uqgb6XPfYDVIcUSS+uCpkOC1/XB9tryuzJ4X317wjCzTCIFzCPu/p9gc9LlE8DdtwBTgZNIrjyOAb5iZkuJdD85zsweJrnyGK1kLHuS7ueocicp8qhypyqVPS3gZ6myJynyGGrZE1bF6gNgoJntY2ZZwNeA50OKJZaeBy4M3l8IPFdp+9fMLNvM9gEGAjOCpsjtZnZYMNPItyodE7ogpvuAz9z9tkq7kiafZpZrZh2D962A44H5JFEe3f0ad+/j7nlE/tZed/dvkkR5bIBkLHuS6ueocic58qhyZy8qexL8Z6myJznyGHrZ4+HN1nEKkRlXFgHXhRVHE+J/DFgDFBOp1X4X6AJMAb4IXjtXSn9dkNfPqTSrCJAPzAn2/RUiizYnwhdwJJFmz9nAx8HXKcmUT2A48FGQxznAr4LtSZPHavkdy5cz5CRlHqP4HrTYskflTnLkU+VO8uUxyu+Dyp4E/lmq7EmOPFbLb9zLHgsOFBERERERkUYKbYFgERERERGRZKGKlYiIiIiISBOpYiUiIiIiItJEqliJiIiIiIg0kSpWIiIiIiIiTaSKVQozszwzm1Nt2/Vm9tMGnGOpmXWtJ821jY1RRJKLyh0RCYPKHokHVawkHlTIiEi8qdwRkTCo7ElhqlhJjcxsqpndYWbvmNkcMxsdbO9iZq+a2Udm9nfAKh3zrJl9aGZzzeySYNskoJWZfWxmjwTbvmlmM4Jtfzez9ODrgeBan5rZVWHkW0TCo3JHRMKgskdiRRUrqUsbdz8C+CHwz2Dbr4G33H0U8DzQr1L677j7wURWqr7czLq4+0Rgt7uPdPdvmNkBwHnAGHcfCZQC3wBGAr3dfai7DwPuj0P+RCTxqNwRkTCo7JEmywg7AAmV17P9MQB3n25m7c2sI3A0cFaw/QUz21zpuMvN7MzgfV9gILCx2rnHAQcDH5gZQCtgPfBfYF8z+wvwAvBqE/IlIolL5Y6IhEFljzQ7VaxS20agU7VtnYElwfvqhZDXsh0zGwscDxzu7rvMbCqQU8M1DXjQ3a+p4RwjgBOBS4Fzge9EkwkRaVFU7ohIGFT2SLNTV8AU5u47gDVmNg7AzDoDJwFvBUnOC7YfCWx1963AdCLN2JjZyXxZSHUANgcFzP7AYZUuVWxmmcH7KcDZZtat/Jpm1j+YZSfN3Z8Gfgkc1CyZFpFQqdwRkTCo7JF4UIuVfAu408xuDT7f4O6LgibrzWb2DtCeL5+k3AA8ZmazgGnA8mD7y8APzGw28DnwXqVr3APMNrNZQZ/jXwCvmlkaUEzkac1u4P5gG8BeT3dEJGmo3BGRMKjskWZl7rV1OZVUFjRr/9TdZ4Ydi4ikBpU7IhIGlT0SK+oKKCIiIiIi0kRqsRIREREREWkitViJiIiIiIg0kSpWIiIiIiIiTaSKlYiIiIiISBOpYiUiIiIiItJEqliJiIiIiIg0kSpWIiIiIiIiTfT/BfnU+KGJNoIAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 1200x400 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "fig, axs = plt.subplots(1, 3, figsize=(12, 4))\n",
        "axs[0].plot(loss_10)\n",
        "axs[0].set_title(\"A2C Loss\")\n",
        "axs[1].plot(rewards_10)\n",
        "axs[1].set_title(\"Episode Reward\")\n",
        "axs[2].plot(regrets_10)\n",
        "axs[2].set_title(\"Episode Regret\")\n",
        "\n",
        "for i in range(3):\n",
        "  axs[i].set_xlabel(\"Updates\")\n",
        "fig.tight_layout()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n2HEY0RypDo7"
      },
      "source": [
        "# Evaluate Generalization\n",
        "\n",
        "Next, we can take the network checkpoint trained on the easy bandit task (0.1 `reward_prob`) and test how it performs on harder tasks:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "executionInfo": {
          "elapsed": 5884,
          "status": "ok",
          "timestamp": 1660493121061,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "Edv3P0sfjORP",
        "outputId": "915c728e-1bf4-4876-ce7c-9dd91df535c8"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(DeviceArray(0.6875, dtype=float32),\n",
              " DeviceArray(1.140625, dtype=float32),\n",
              " DeviceArray(3.5968752, dtype=float32))"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "env_params_10 = EnvParams(reward_prob=0.1)\n",
        "env_params_25 = EnvParams(reward_prob=0.25)\n",
        "env_params_40 = EnvParams(reward_prob=0.4)\n",
        "\n",
        "rng_batch = jax.random.split(rng, 64)\n",
        "_, _, _, _, regret_10_10 = batch_rollout_fn(net_params_10, rng_batch, 0.8, env_params_10, 100)\n",
        "_, _, _, _, regret_10_25 = batch_rollout_fn(net_params_10, rng_batch, 0.8, env_params_25, 100)\n",
        "_, _, _, _, regret_10_40 = batch_rollout_fn(net_params_10, rng_batch, 0.8, env_params_40, 100)\n",
        "\n",
        "regret_10_10.mean(), regret_10_25.mean(), regret_10_40.mean()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BI2AatHTEcgm"
      },
      "source": [
        "Even though the meta-learner was not trained on hard bandit tasks, its meta-learning exploration algorithm is capable of generalizing to the unseen setting. Let's now train agents on harder bandit tasks:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "executionInfo": {
          "elapsed": 171802,
          "status": "ok",
          "timestamp": 1660493292968,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "HW9aiATzdct_",
        "outputId": "fd603aae-5ee1-40ad-81e6-50e535a98d20"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "# Updates: 0 | Loss: 4.183656215667725 | Return 49.75  | Regret: 25.0390625 | b_e: 0.9990000128746033\n",
            "# Updates: 200 | Loss: -0.1636858582496643 | Return 50.265625  | Regret: 25.125 | b_e: 0.8178321719169617\n",
            "# Updates: 400 | Loss: -0.07612765580415726 | Return 57.40625  | Regret: 17.421875 | b_e: 0.6695191860198975\n",
            "# Updates: 600 | Loss: -0.0006994768045842648 | Return 60.234375  | Regret: 15.28125 | b_e: 0.5481024384498596\n",
            "# Updates: 800 | Loss: 0.05531803518533707 | Return 61.46875  | Regret: 14.234375 | b_e: 0.4487048387527466\n",
            "# Updates: 1000 | Loss: 0.07975044846534729 | Return 63.0625  | Regret: 11.875 | b_e: 0.3673325479030609\n",
            "# Updates: 1200 | Loss: 0.13205844163894653 | Return 65.234375  | Regret: 10.2734375 | b_e: 0.3007172644138336\n",
            "# Updates: 1400 | Loss: 0.2039772868156433 | Return 67.859375  | Regret: 7.0703125 | b_e: 0.2461823672056198\n",
            "# Updates: 1600 | Loss: 0.21568433940410614 | Return 69.75  | Regret: 5.453125 | b_e: 0.20153741538524628\n",
            "# Updates: 1800 | Loss: 0.2601448595523834 | Return 71.125  | Regret: 3.5625 | b_e: 0.16498881578445435\n",
            "# Updates: 2000 | Loss: 0.28076231479644775 | Return 73.171875  | Regret: 2.875 | b_e: 0.13506826758384705\n",
            "# Updates: 2200 | Loss: 0.3188568353652954 | Return 73.375  | Regret: 1.8359375 | b_e: 0.11057380586862564\n",
            "# Updates: 2400 | Loss: 0.33819934725761414 | Return 73.234375  | Regret: 1.453125 | b_e: 0.09052134305238724\n",
            "# Updates: 2600 | Loss: 0.36133426427841187 | Return 73.046875  | Regret: 1.40625 | b_e: 0.074105404317379\n",
            "# Updates: 2800 | Loss: 0.3518439829349518 | Return 73.90625  | Regret: 1.3125 | b_e: 0.060666490346193314\n",
            "# Updates: 3000 | Loss: 0.364878386259079 | Return 73.96875  | Regret: 1.109375 | b_e: 0.0496646910905838\n",
            "# Updates: 3200 | Loss: 0.3652738928794861 | Return 73.875  | Regret: 1.03125 | b_e: 0.040658049285411835\n",
            "# Updates: 3400 | Loss: 0.352735698223114 | Return 74.625  | Regret: 1.0390625 | b_e: 0.0332847461104393\n",
            "# Updates: 3600 | Loss: 0.35522526502609253 | Return 73.75  | Regret: 0.7578125 | b_e: 0.027248600497841835\n",
            "# Updates: 3800 | Loss: 0.36759716272354126 | Return 73.96875  | Regret: 1.078125 | b_e: 0.02230709046125412\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "(DeviceArray(0.9624999, dtype=float32),\n",
              " DeviceArray(1.2421875, dtype=float32),\n",
              " DeviceArray(2.3500004, dtype=float32))"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "net_params_25, loss_25, rewards_25, regrets_25 = run_a2c_training(rng, env_params_25, verbose=True)\n",
        "\n",
        "_, _, _, _, regret_25_10 = batch_rollout_fn(net_params_25, rng_batch, 0.9, env_params_10, 100)\n",
        "_, _, _, _, regret_25_25 = batch_rollout_fn(net_params_25, rng_batch, 0.9, env_params_25, 100)\n",
        "_, _, _, _, regret_25_40 = batch_rollout_fn(net_params_25, rng_batch, 0.9, env_params_40, 100)\n",
        "\n",
        "regret_25_10.mean(), regret_25_25.mean(), regret_25_40.mean()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "executionInfo": {
          "elapsed": 173375,
          "status": "ok",
          "timestamp": 1660493466445,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "mjpnjTqV6Tz1",
        "outputId": "10117bcd-e8ca-47ff-e03a-6886db2d72f5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "# Updates: 0 | Loss: 4.186663627624512 | Return 49.8125  | Regret: 10.009376525878906 | b_e: 0.9990000128746033\n",
            "# Updates: 200 | Loss: -0.15455523133277893 | Return 50.984375  | Regret: 10.05312728881836 | b_e: 0.8178321719169617\n",
            "# Updates: 400 | Loss: -0.03203165903687477 | Return 50.015625  | Regret: 10.00625228881836 | b_e: 0.6695191860198975\n",
            "# Updates: 600 | Loss: 0.10917842388153076 | Return 50.359375  | Regret: 9.850001335144043 | b_e: 0.5481024384498596\n",
            "# Updates: 800 | Loss: 0.0710330531001091 | Return 49.875  | Regret: 10.212502479553223 | b_e: 0.4487048387527466\n",
            "# Updates: 1000 | Loss: 0.12697479128837585 | Return 48.84375  | Regret: 10.212501525878906 | b_e: 0.3673325479030609\n",
            "# Updates: 1200 | Loss: 0.19187688827514648 | Return 50.734375  | Regret: 9.987502098083496 | b_e: 0.3007172644138336\n",
            "# Updates: 1400 | Loss: 0.2452111542224884 | Return 52.359375  | Regret: 8.06875228881836 | b_e: 0.2461823672056198\n",
            "# Updates: 1600 | Loss: 0.23364955186843872 | Return 53.453125  | Regret: 7.478126049041748 | b_e: 0.20153741538524628\n",
            "# Updates: 1800 | Loss: 0.25738757848739624 | Return 53.421875  | Regret: 6.484375953674316 | b_e: 0.16498881578445435\n",
            "# Updates: 2000 | Loss: 0.27192187309265137 | Return 54.890625  | Regret: 6.028126239776611 | b_e: 0.13506826758384705\n",
            "# Updates: 2200 | Loss: 0.33167412877082825 | Return 54.15625  | Regret: 5.85312557220459 | b_e: 0.11057380586862564\n",
            "# Updates: 2400 | Loss: 0.3137226104736328 | Return 54.25  | Regret: 5.075000762939453 | b_e: 0.09052134305238724\n",
            "# Updates: 2600 | Loss: 0.3329048752784729 | Return 55.453125  | Regret: 4.084375381469727 | b_e: 0.074105404317379\n",
            "# Updates: 2800 | Loss: 0.345061719417572 | Return 56.21875  | Regret: 4.443750858306885 | b_e: 0.060666490346193314\n",
            "# Updates: 3000 | Loss: 0.3961333632469177 | Return 56.09375  | Regret: 3.44687557220459 | b_e: 0.0496646910905838\n",
            "# Updates: 3200 | Loss: 0.3690904676914215 | Return 56.90625  | Regret: 3.325000762939453 | b_e: 0.040658049285411835\n",
            "# Updates: 3400 | Loss: 0.3843540847301483 | Return 57.71875  | Regret: 3.0656256675720215 | b_e: 0.0332847461104393\n",
            "# Updates: 3600 | Loss: 0.3750694692134857 | Return 57.0625  | Regret: 2.2562503814697266 | b_e: 0.027248600497841835\n",
            "# Updates: 3800 | Loss: 0.39970308542251587 | Return 57.546875  | Regret: 2.609375476837158 | b_e: 0.02230709046125412\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "(DeviceArray(1.1499999, dtype=float32),\n",
              " DeviceArray(1.28125, dtype=float32),\n",
              " DeviceArray(2.0656252, dtype=float32))"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "net_params_40, loss_40, rewards_40, regrets_40 = run_a2c_training(rng, env_params_40, verbose=True)\n",
        "\n",
        "_, _, _, _, regret_40_10 = batch_rollout_fn(net_params_40, rng_batch, 0.95, env_params_10, 100)\n",
        "_, _, _, _, regret_40_25 = batch_rollout_fn(net_params_40, rng_batch, 0.95, env_params_25, 100)\n",
        "_, _, _, _, regret_40_40 = batch_rollout_fn(net_params_40, rng_batch, 0.95, env_params_40, 100)\n",
        "\n",
        "regret_40_10.mean(), regret_40_25.mean(), regret_40_40.mean()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "height": 313
        },
        "executionInfo": {
          "elapsed": 511,
          "status": "ok",
          "timestamp": 1660493467693,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "GftNhxQt_6jH",
        "outputId": "6e578759-059e-4b85-e71b-9f787ab45891"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Text(0.5, 1.0, 'Wang et al. F2f) - Cumulative Regret')"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVcAAAEWCAYAAADSNdTRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAjX0lEQVR4nO3debxcVZnu8d9zkgCRIBESBpnStjgRGYOCIBdo5QIyOHARRBFsG6VxQKW5NG1LQPHqRUQxCkYFZJBBQYgYVBxCQGUIEAIhKChBAjGQSIBABJK8/cdaZSqVc6r2Oal9qvbJ881nf07tae1VlTrvWfvda6+tiMDMzNqrp9MVMDMbihxczcxK4OBqZlYCB1czsxI4uJqZlcDB1cysBA6uhqSLJH1hEI5zvKQFkpZI2lTSA5I2Kfu4g0XSXElvG+C+b5X0h3bXyTrHwbWOpP+UNLVh2YN9LDticGvXN0nTJH14kI51kaQXc4CsTe+VtK6k70l6RNKzku6WdEDdfiOArwL7RcSoiFgAXAD83zWsz8slfU3SX3JdHsrzY9bsnZZLUkh6dW0+Im6OiNeWcJyJkl7Kn81iSb+TtHu7j1OwLoPyR7xbOLiuajqwh6RhAJI2A0YAOzcse3Xedm31/3OArE1XAsOBR4H/BWwI/DdwlaRxeZ9NgfWA2XXl/AD4oKR1B1IJSesAvwK2A/YHXg68BVgEvGkgZQ5RV0bEKGAM8Bvgh+0+gBLHkzr+MFZ1BymY7pjn9yJ9Gf/QsOxPEfG4pGMlzckttT9L+kitIEl7S5on6TOSnpA0X9Kxdes3lvQTSc9IukPSFyTd0lfFJO2WWx2LJd0jae+8/EzgrcCk3DqZ1Mf+P5T0V0lPS5ouabuBfUS9i4jnImJiRMyNiBURcT3wMLCLpNeQPkOAxZJ+nfeZBzwF7DbAwx4NbA28KyLuz8d9IiI+HxFTYfUWYn3rqe7/6OS6/6N3SjpQ0h8l/U3Sqb3tW79/bxWT9CZJv8//X/MlTcp/DJBU+8N8T13L/x9lSTpF0o8ayvu6pHPz6w3zWcJ8SY/l786wVh9WRCwDLgO2kDS2VVmShkk6W9JCSQ9L+lj+PIfn9dMknSnpt8DzwKskvU7Sjfmz+4Okw/O2xwFHASfn9/yTVvWtOgfXOhHxInAbKYCSf94M3NKwrPbL8QRwEKnFdCxwjqSd64rcjNSK2wL4V+Cbkl6R130TeC5v88E89UrSFsBPgS8AGwEnAVdLGhsR/5Xr+LHcivxYH8XcAGwLbALcRfolK42kTYHXALMj4o+k1iXA6IjYt27TOcAOAzzM24CfRcSSgdeUzUgt6i2AzwHfAd4P7EL6o/U5Sa8aQLnLgU+RWou7A/8C/DtARNS+SzvUtfzrXQ4cKOnlkIIccDippQ/wfWAZ6QxqJ2A/oGVaKAf3o0kt+6cKlPVvwAGkhsXOwDt7KfYDwHHABsCTwI25npsARwLfkrRdREwmfedqZz0Ht6pv1Tm4ru4mVgbSt5IC180Ny24CiIifRsSfIrkJ+EVeX/MScEZEvJRbUkuA1+ZflvcAp0XE8xFxP+lL3pf3A1MjYmpund0IzAAOLPqmIuKCiHg2Il4AJgI7SNqw6P4NTsotssWSFjauVMqvXgZ8PyIeaFHWs8DoAdZjY2D+APeteQk4MyJeAq4gBcOv589qNimNsX1/C42IOyPi1ohYFhFzgW+TUiZF9n2E9AfwnXnRvsDzEXFr/qN1AHBiPlt4AjgHaHYN4HBJi4GlpIB5WEQsK1DW4aTPYl5EPAV8qZeyL4qI2blVvD8wNyIuzO/7LuBq4LAi73uocXBd3XRgz9zCHBsRDwK/A96Sl43P2yDpAEm35lOgxaRgV38hZVH+0tU8D4wCxrIyR1lT/7rRNsD/qQtoi4E9gc2LvKF8evclSX+S9AwwN68a6EWfr0TE6DytUoZS3u0S4EWgr1Z0vQ2AxX3Uu/6i2da9bLKIgp9BE4siYnl+vTT/XFC3finp/6xfJL1G0vU5FfMM8EX693n/gNTyA3gfK1ut25BSV/PrvgvfJrUU+3JVRIwm5b3vI7XKi5T1Slp/R+uXbQO8ueF7ehTp7GCtM7zTFehCvyedyh8H/BYgIp6R9Hhe9nhEPKx0EeZq0mnWdRHxkqRrARU4xpOkU7EtgT/mZVs12f5R4JKI+Lc+1rca2ux9wKGk0+i5pPf3VMG6FiZJwPdIv8QH5tZgK68Hzu5tRb4I08wvgS9IWj8inutjm+eBl9XNbwb0mict4LleyurLecDdwJER8aykE+lfC+6HwNmStgTeRUotQPouvACMafjD3VJELFS6LnCHpB8UKGs+6Tta09t3tP679yhwU0S8va8q9Ke+VeeWa4OIWEo65f40KR1Qc0teVsu3rgOsSw6USt2O9it4jOXANcBESS+T9DpSkO7LpcDBkv53boWuly+A1L74C4BmecENSL9Ei0jB4YtF6jkA55GC5cH5c2wq55I3Am4d4PEuIf1CX50vpPQoXSg8VVItZTITeF/+3Pan4Kl5H2aScqEbKfUaObHJthsAzwBL8v/v8Q3rm/6fRcSTwDTgQuDhiJiTl88npZ/OVuqG1iPpnyUVTTk8APwcOLlAWVcBn5S0haTRtO42dz3wGkkfkDQiT7tKen2R9zzUOLj27ibSqVH91fub87LpABHxLPAJ0hfwKVLrcEo/jvExUgvyr6QgcTkpAK4mIh4ltTxPJQXzR4H/YOX/39eBwyQ9Vbui3OBi4BHgMeB+mgQzSVs3OQ3vk6RtgI+QLn78te50/qgmu72PlJft9X23kvd7G/AA6ULKM8DtpNPv2/JmnwQOJqUejgKuHcixskuAe0it/18AjRei6p1Een/Pki6SNW47Efh+Pn0+vI8yfkB6fz9oWH406Y/7/aTv3o/oX3rkLOA4pRs4mpX1HdL7nEVqhU8lnXEtbywQ/vE7sR8pZ/s46bv9ZVIjBNJZzRvye762H/WtJHmw7O4g6cvAZhHRZ6+BoSSnVe4B9soXUqzL5bOz8yNim07XpQrccu2QfBq7vZI3kbpq/bjT9RosEfFCRLzOgbV7SRqp1Od3eE7hnMZa9B1dUw6unbMBKe/6HCm1cDZwXUdrZLYqAaeT0gV3k/okf66jNaoQpwXMzErglquZWQncz7XBmDEbx7it+3WhfO3y4t87XYOu98jsP7beaC23kBULI2LsQPffSsPj7wW7zS5kxc8jYv+BHmugHFwbjNt6a2bcMq3T1ehaK/4yp9NV6Hr//vq++tBbzbdZ8sia7P93gvewfsFjPduR4ScdXM2sckT35zQdXM2scgQMV8G7tzt0zb7bg7+ZWa96VGxqJd9OfrvSOMmzJZ3eyzZ7K42FPDNPLbukueVqZpXUxpbhC8C+EbEkD5d5i6QbIqLxNvGbI+KgooU6uJpZ5QjRUzQt0EKkzv61AddH5GmNkwlOC5hZJfUUnIrIo6bNJD1d5MaIuK2XzXbPqYMbVOAxSW65mlnliGL51GyMpBl185PzY2f+IQ8DumMeWvHHksZHxH11m9wFbJNTBweSRlfbttlBHVzNrHoEw4qnBRZGxIQiG0bEYknTSI+sua9u+TN1r6dK+pakMRGx2mOOapwWMLPKqfVzbUdaQNLY3GJF0khWjhFcv81m+Ukb5FHsekiDz/fJLVczq6R+pAVa2Zw0cPkwUtC8KiKul/RRgIg4n/SInuMlLSM9V+2IaDHqlYOrmVVSu067I2IW6bHijcvPr3s9CZjUn3IdXM2sctIFrbY+X7PtHFzNrHLS7a+drkVzDq5mVkndfjXewdXMKqmH7m66OriaWeX08yaCjnBwNbNKclrAzKzNVHA4wU5ycDWzSio8WHaHOLiaWeX4MS9mZiVxWsDMrM2E3BXLzKwMbrmambWZgGEOrmZm7ee0gJlZm7mfq5lZSdwVy8ysBF3ecHVwNbPq8WDZZmYlcVrAzKwE3d1u7fLgKmk5cG/doisi4kudqo+ZdQ85LbBGlkbEjp2uhJl1F9H9LdduT1v0StLnJN0h6T5Jk5X/hEn6hKT7Jc2SdIWkHkkPShqb1/dIekjSmM6+AzNbUz0Fp07p9uA6UtLMuum9efmkiNg1IsYDI4GD8vJTgJ0iYnvgoxGxArgUOCqvfxtwT0QsHMw3YWbtJxWbOqXbg+vSiNixbroyL99H0m2S7gX2BbbLy2cBl0l6P7AsL7sAODq//hBwYeNBJB0naYakGU8uXFTeuzGztkjjuarQ1CndHlxXI2k94FvAYRHxRuA7wHp59TuAbwK7AHdKGh4RjwILJO0LvBm4obHMiJgcERMiYsLYMRsPyvswszWjglOnVC64sjKQLpQ0CjgMUj4V2CoifgOcDIwGRuVtv0tKD1wVEcsHt7pmVoYeFZs6pdt7C4yUNLNu/mcRcYqk75C6aM0F7sjrhgGXStqQ9AfrnIhYnNdNIaUDVksJmFkVCXV5f4GuDq4RMayP5Z8FPtvLqj37KGoH0oWsB9pVNzPrnE6f8hdRxbRAv0g6Bbga+M9O18XM2qRgSqBIWkDSepJul3SPpNmSTu9lG0k6N3flnCVp51bldnXLtR3yHV2+q8tsiGljT4AXgH0jYomkEcAtkm6IiFvrtjkA2DZPbwbOyz+b1M/MrGKK9hQoEn4jWZJnR+QpGjY7FLg4b3srMFrS5s3KdXA1s0rqx00EY2r92PN03OplaVi+eP4EcGNE3NawyRbAo3Xz8/KyPg35tICZDU39SAosjIgJzTbIXTR3lDQa+LGk8RFxX4vDNbZuV+GWq5lVkgr+64/cfXMasH/DqnnAVnXzWwKPNyvLwdXMKqf2aO0iU8uypLG5xYqkkaQxSBq7bU4Bjs69BnYDno6I+c3KdVrAzCqpjf1cNwe+L2kYqcF5VURcL+mjABFxPjAVOBB4CHgeOLZVoQ6uZlZJ7bpDKyJmATv1svz8utcBnNCfch1czaySuvxBBA6uZlY9acjB7ubgamaV1OUNVwdXM6umni7PCzi4mlnlVGFULAdXM6seyY/WNjMrQyefMlCEg6uZVZK6PLo6uJpZ5UjQ0+V9sRxczaySnHM1MytBl8dWB1czqya3XM3M2ky45Wpm1n7yHVpmZiUQPe6KZWbWXgLkrlhmZm0mX9AyMytFl8dWB1czqya3XM3MStDlsdXB1cyqR4Jh7i1gZtZ+TgtUzfKXiMULOl2LrhXPP9PpKnS9sw8Z3+kqdL1vT7l1jcvo8tjaOrhK2gOYCGyTtxfpMd6vKrdqZma9Gyq3v34P+BRwJ7C83OqYmRUgDYnBsp+OiBtKr4mZWT8MhQtav5F0FnAN8EJtYUTcVVqtzMyaGCppgTfnnxPqlgWwb/urY2ZWTOV7C0TEPoNRETOzwtS+lqukrYCLgc2AFcDkiPh6wzZ7A9cBD+dF10TEGc3KLdJbYEPgNGCvvOgm4IyIeLof9Tcza6s2tlyXAZ+JiLskbQDcKenGiLi/YbubI+KgooUWGbTrAuBZ4PA8PQNcWPQAZmZlkIpNrUTE/No1pIh4FpgDbLGm9SuSc/3niHhP3fzpkmau6YHNzAZKgp5hhVuuYyTNqJufHBGTey9X44CdgNt6Wb27pHuAx4GTImJ2s4MWCa5LJe0ZEbfkg+8BLC2wn5lZSdSftMDCiJjQaiNJo4CrgRMjovFWxLuAbSJiiaQDgWuBbZuVVyS4Hg98P+deBfwNOKbAfmZm5WljP1dJI0iB9bKIuKZxfX2wjYipkr4laUxELOyrzCK9BWYCO0h6eeNBzMw6pk0XtJSawN8D5kTEV/vYZjNgQUSEpDeRrlctalZun8FV0vsj4lJJn25YDkBflTAzK117H/OyB/AB4N6660mnAlsDRMT5wGHA8ZKWkdKiR0RENCu0Wct1/fxzg17WNS3UzKxcgmHteUJhvp7UNFJHxCRgUn/K7TO4RsS388tfRsRv69fli1pmZh0h0fUDtxQJ/d8ouMzMbPC0q6NrSZrlXHcH3gKMbci7vhwYVnbFzMya6faWa7Oc6zrAqLxNfd71GVJy18ysc6o6cEtE3ATcJOmiiHhkEOtkZtac1NZ+rmVolhb4WkScCEyStFrvgIg4pMyKmZk1ozb1FihLs7TAJfnnVwajImZmhVVgtOxmaYE788+bBq86ZmbFqLsbrk3TAvfS5GaBiNi+lBqZmRVR1ZYrUBsU9oT8s5YmOAp4vrQamZm1UuWnv9Z6CEjaIyLq78g6RdJvgaaPODAzK1WXt1yLZC3Wl7RnbUbSW1g57oCZ2aCTUm+BIlOnFBnP9V+BC/J4rgCLgQ+VViMzsyKqmhaoyb0GauO5yg8mNLPO6+y4AUUUefrrusB7gHHA8LrxXJ1zNbOOaeN4rqUokha4DngauBN4odzqmJkVIKqfFgC2jIj9S6+JmVk/dPvtr0Vq9ztJbyy9JmZmRRUdy7WDqYMiwXVP4E5Jf5A0S9K9kma12klSSLqkbn64pCclXd+fCkqaJmlCfj1V0uj+7G9mQ5N6VGjqlCJpgQMGWPZzwHhJIyNiKfB24LEBlgVARBy4Jvub2RDS5Re0WrZc851ao4GD8zS6H+O73gC8I78+Eri8tkLS+pIukHSHpLslHZqXj5R0RW4lXwmMrNtnrqQxksZJuq9u+UmSJubX0ySdI2m6pDmSdpV0jaQHJX2hYL3NrJvVLmgVmTqkZXCV9EngMmCTPF0q6eMFy78COELSesD2wG116/4L+HVE7ArsA5wlaX3geOD5PDDMmcAuRd9MnRcjYi/gfFJvhxOA8cAxkjbu5T0eJ2mGpBlPLnpqAIczs8EmqdDUKUXv0HpzRDwHIOnLwO8p8JDCiJglaRyp1Tq1YfV+wCGSTsrz65GeE74XcG7d/i3zu72Ykn/eC8yOiPm57n8GtgIWNdRzMjAZYMIO4/3YcLOu175Ha5elSHAVsLxufjktnvHdYAppwO29gfpWo4D3RMQfVjlY+kvTKsAtY9VW93oN62v9cVewat/cFRR7z2bWzSowWHaR0H8hcJukiTmveSvwvX4c4wLgjIi4t2H5z4GPK0dTSTvl5dNJwxoiaTwpndBoAbCJpI3zHWQH9bKNmQ1lXd4Vq8jYAl+VNI3UJUvAsRFxd9EDRMQ84Ou9rPo88DVgVg6wc0lB8jzgwpwOmAnc3kuZL0k6g5TDfRh4oGh9zGwoEPRUNC0gaVdgTETcEBF3AXfl5YdI6qk9BqYvETGql2XTgGn59VLgI71ssxQ4oo8yx9W9Ppecm23YZu/ejte4zswqrsJpgbOAOb0svz+vMzPrjFrOtaJpgY0jYm7jwoh4qLfuTGZmg0cwbFinK9FUs5bryCbr/CQCM+usNrVcJW0l6Tf5pqPZuW9/4zaSdK6kh/INTju3KrdZcP2lpDNrV/PrDnI68OuWNTYzK0t70wLLgM9ExOuB3YATJL2hYZsDgG3zdBzpwntTzdICnwG+CzwkaWZetgMwA/hwkRqbmZWmTfnUfJPR/Pz6WUlzgC1I15dqDgUujogAbpU0WtLmtRuUetPs6a/PAUdKehWwXV48OyL+vIbvxcxsDfWrK9YYSTPq5ifnuzJXLzXdUboTq96qDynYPlo3Py8v639wrcnB1AHVzLqH6E9wXRgRE1oWKY0CrgZOjIhnejlio6Z3kvpWUDOrpjZ2s5I0ghRYL4uIa3rZZB5pXJKaLYHHm5XZ3bc4mJn1Qgj19BSaWpaVLtp/D5gTEV/tY7MpwNG518BuwNPN8q1QsOUqaRiwaf32EfGXIvuamZWifS3XPYAPAPfWXbw/lTRKHxFxPmlUvwOBh4DngWNbFVrk0dofB04jDZayIi8Oeh9QxcysfG0cFSsibqHFSH+5l8AJ/Sm3SMv1k8BrI2JRyy3NzAZLl48tUCS4Pgo8XXZFzMyK6/7bX4sE1z8D0yT9lLqBp5skfs3MylWBwbKLBNe/5GmdPJmZdV7Vg2tEnD4YFTEzK67ag2V/LSJOlPQTerkTISIOKbVmZmbNVLjlekn++ZXBqIiZWWFVzrnWHuMSETcNXnXMzIoYAr0FJG0L/D/gDdQ9wjoiXlVivczMmuvylmvRR2ufRxpQdh/gYlamDMzMBl8FnqFVJLiOjIhfAYqIRyJiIrBvudUyM2sm9xYoMnVIkX6uf5fUAzwo6WPAY8Am5VbLzKyFIZAWOBF4GfAJYBfg/cAHS6yTmVlzAnqGFZs6pGnLNQ81eHhE/AewhALDbJmZlU/Q090t12Y3EQyPiGWSdpGkPOSWmVl3UEXv0AJuB3YG7gauk/RD4Lnayj4ehWBmNji6POda5ILWRsAiUg+BIGU7AnBwNbPOUIXHFgA2kfRp4D5WBtUapwjMrLMq3HIdBoxiAI+UNTMrXQd7AhTRLLjOj4gzBq0m3WL5MuLphZ2uRfd66cVO16DrrfvqV3a6CkNfxdMC3d3mNrO1W4XTAv8yaLUwM+uvqnbFioi/DWZFzMwKU4VvIjAz62oVvqBlZtalVN20gJlZ1xJOC5iZlaLCvQXMzLpXl6cFurt2Zma9qfUWKDK1LEoXSHpC0n19rN9b0tOSZubpc0Wq6JarmVVT+3oLXARMIj0fsC83R8RB/SnUwdXMKqh9vQUiYrqkcW0prI7TAmZWPbXeAsXSAmMkzaibjhvAEXeXdI+kGyRtV2QHt1zNrJqKt1wXRsSENTjSXcA2EbFE0oHAtcC2rXZyy9XMqkkqNq2hiHgmIpbk11OBEZLGtNrPLVczq6DBG3JQ0mbAgogISW8iNUoXtdrPwdXMqqf2aO12FCVdDuxNys3OA04DRgBExPnAYcDxkpYBS4Ejijyw1cHVzCqoPaf8ABFxZIv1k0hdtfrFwdXMqqnCTyIwM+tOwmMLmJm1n4ccNDMrhwfLNjNrMz/mxcysJE4LmJmVwBe0zMzazRe0zMxKIbdczczaTIKe7g5f3V07M7O+uLeAmVkJnHM1M2sz3/5qZlYG9xYwMytHl7dcuzb0S1rSMH+MpH6PqdhQxtwij2cwsy4nwbBhxaYOGbItV0nDI2JZp+thZiVxWqD9JB0MfBZYh/Qsm6MiYoGkicArgXHAQkkfBy4HxgK3k9LgZjYUdHlaoJuD60hJM+vmNwKm5Ne3ALvlB4Z9GDgZ+ExetwuwZ0QslXQucEtEnCHpHcBAnlduZl3HF7TWxNKI2LE2I+kYoPbs8S2BKyVtTmq9Ply335SIWJpf7wW8GyAifirpqd4OJOk4cuDdevNN2/gWzKw0Xd5y7e7Q37dvAJMi4o3AR4D16tY917Bty6c0RsTkiJgQERPGbjS6fbU0s3JIMGx4salDqhpcNwQey68/2GS76cBRAJIOAF5Rcr3MbJBIKjR1SlWD60Tgh5JuBhY22e50YC9JdwH7AX8ZhLqZ2WBQT7GpQ7o25xoRoxrmLwIuyq+vA67rZZ+JDfOLSEG15lNtrqaZdYJvfzUzK4N7C5iZlcMtVzOzNqvd/trFHFzNrJq6PC3Q3bUzM+uLVGxqWYwukPSEpPv6WC9J50p6SNIsSTsXqZ6Dq5lVlApOLV0E7N9k/QHAtnk6DjivSKEOrmZWQQVbrQVarhExHfhbk00OBS6O5FZgdL71vinnXM2smor3FhgjaUbd/OSImNyPI20BPFo3Py8vm99sJwdXM6se0Z8LWgsjYkLrzZoerVHLMUscXM2smgavm+s8YKu6+S2Bx1vt5JyrmVVU2y5otTIFODr3GtgNeDoimqYEwC1XM6ukYherCpUkXQ7sTcrNzgNOA0YARMT5wFTgQOAh4Hng2CLlOriaWTW1KbhGxJEt1gdwQn/LdXA1s2rq8ju0HFzNrKI8cIuZWXsVvEGgkxxczayaHFzNzMrg4Gpm1nadfPhgEQ6uZlZBfsyLmVk53HI1M2szP/3VzKwsDq5mZu3nlquZWQm6O7Y6uJpZFbm3gJlZ+/mClplZWRxczczazy1XM7N286hYZmbl6PILWkpPMLAaSU8Cj3S6HnXGAAs7XYku58+ouW78fLaJiLED3VnSz0jvq4iFEbH/QI81UA6uXU7SjDV85vqQ58+oOX8+ndHd7Wozs4pycDUzK4GDa/eb3OkKVIA/o+b8+XSAc65mZiVwy9XMrAQOrmZmJXBw7RBJyyXNrJtO6XSdBpOkkHRJ3fxwSU9Kur6f5UyTNCG/nippdJur2pUkLWmYP0bSpDUsc66kon1HrQXfodU5SyNix05XooOeA8ZLGhkRS4G3A4+tSYERcWBbarYWkDQ8IpZ1uh5DmVuuXUbS5yTdIek+SZOVnx8s6ROS7pc0S9IVknokPShpbF7fI+mhirU8bgDekV8fCVxeWyFpfUkX5M/ibkmH5uUj8/ufJelKYGTdPnMljZE0TtJ9dctPkjQxv54m6RxJ0yXNkbSrpGvyZ/mFQXjPpZN0sKTb8uf2S0mb5uUT83fqF8DFkjaW9Iu83bfp9mGmKsbBtXNGNqQF3puXT4qIXSNiPClwHJSXnwLsFBHbAx+NiBXApcBRef3bgHsiottuc2zmCuAISesB2wO31a37L+DXEbErsA9wlqT1geOB5/PncCawywCO+2JE7AWcD1wHnACMB46RtPGA383gWuX7A5xRt+4WYLeI2In0GZ9ct24X4NCIeB9wGnBL3m4KsPXgVH3t4LRA5/SVFthH0snAy4CNgNnAT4BZwGWSrgWuzdteQAoOXwM+BFxYao3bLCJmSRpHarVObVi9H3CIpJPy/HqkX/69gHPr9p81gENPyT/vBWZHxHwASX8GtgIWDaDMwbbK90fSMUDtFtctgSslbQ6sAzxct9+UnIaB9Fm+GyAifirpqbIrvTZxy7WL5Bbct4DDIuKNwHdIQQXS6fM3SS2PO3PO7FFggaR9gTeTTrOrZgrwFepSApmA90TEjnnaOiLm5HWtOmcvY9Xv9noN61/IP1fUva7ND4UGxzdIZ0BvBD7Cqu//uYZt3dG9JA6u3aX2S7BQ0ijgMEj5VGCriPgN6RRvNDAqb/tdUnrgqohYPrjVbYsLgDMi4t6G5T8HPl6Xc94pL59OToVIGk9KJzRaAGySc4rrsjK1srbYkJUXBz/YZLv6z/IA4BUl12utMhT+SlfVyJwrq/lZRJwi6Tuk09W5wB153TDgUkkbklp050TE4rxuCikdUKmUQE1EzAO+3suqz5PSHbNygJ1LCpLnARfmdMBM4PZeynxJ0hmkHO7DwANl1L2LTQR+KOkx4Fbgn/rY7nTgckl3ATcBfxmc6q0dfPtrxeU+nudExFs7XRczW8kt1wrLNx4cz8oeA2bWJdxyNTMrgS9omZmVwMHVzKwEDq5mZiVwcLUByX1Ia7df/lXSY3Xz6xTYf29Jb2my/gBJM/L9/w9I+kqb6n2RpFr/4e9KekN+fWrDdr9rx/Fs7eXeAjYgEbEI2BHSgCDAkojoTwDcG1gCrBbE8s0Bk4B3RMQDkoYDx61hlVcTER+umz0V+GLduj4Dv1kRbrla20jaRdJNku6U9PN8b3tvI3qNAz4KfCq3dBv76J4MnBkRDwBExLKI+FYuaxtJv8pl/UrS1nn5RZLOlfQ7SX+ua51K0qR8/J8Cm9TVd5qkCZK+xMqBUC7L65bU7X+W0ihl99YG2Mkt72mSfpRb1pfV7iYzA7dcrX1Euqf90Ih4MgehM0kDypwC/FNEvCBpdEQslnQ+fbd2xwNn93GcScDFEfF9SR8iDeLyzrxuc2BP4HWkO9d+BLwLeC3wRmBT4H7SLbf/kO+M+1gfA+m8m9RC3wEYA9whaXpetxOwHfA48FtgD9KIVGZuuVrbrEsKijfm23o/SxqdCVaO6PV+0qAqa2J34Af59SWkYFpzbUSsiIj7SYEU0shPl0fE8oh4HPh1P4+3Z93+C0i3ie6a190eEfPy8I8zgXH9fjc2ZLnlau0i0vB9u/ey7h2kIHcI8N+StmtR1mzS6F/3FDhu/V0w9SNcqY9t+qvZqX798Zbj3yer45artcsLwFhJuwNIGiFpuyYjej0LbNBHWWcBp0p6TS6rR9Kn87rfAUfk10fR+jR8OmlA7mE5B7xPH9u9JGlEH/u/N+8/lvRHYrXBYswaObhau6wgDZH4ZUn3kE6T38LKEb3uBe5m5YhePwHe1dsFrYiYBZxIGrFpDnAfKZ8K8Ang2Dwq1geAT7ao14+BB0kjjZ1HOq3vzWTSCFyX9bL/LFIr+tfAyRHx1xbHNPPYAmZmZXDL1cysBA6uZmYlcHA1MyuBg6uZWQkcXM3MSuDgamZWAgdXM7MS/A9G5G4mY3N/vQAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 600x400 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "results = jnp.array([[regret_10_10.mean(), regret_10_25.mean(), regret_10_40.mean()],\n",
        "                     [regret_25_10.mean(), regret_25_25.mean(), regret_25_40.mean()],\n",
        "                     [regret_40_10.mean(), regret_40_25.mean(), regret_40_40.mean()]])\n",
        "\n",
        "fig, ax = plt.subplots()\n",
        "img = ax.imshow(results, cmap='Reds')\n",
        "\n",
        "label_list = ['Easy', 'Medium', 'Hard']\n",
        "ax.set_xticks([0.0, 1.0, 2.0])\n",
        "ax.set_xticklabels(label_list)\n",
        "ax.set_xlabel('Test Condition')\n",
        "\n",
        "ax.set_yticks([0.0, 1.0, 2.0])\n",
        "ax.set_yticklabels(label_list)\n",
        "ax.set_ylabel('Train Condition')\n",
        "fig.colorbar(img)\n",
        "ax.set_title(\"Wang et al. F2f) - Cumulative Regret\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "executionInfo": {
          "elapsed": 1,
          "status": "ok",
          "timestamp": 1660493467850,
          "user": {
            "displayName": "Robert Lange",
            "userId": "17173340079122882032"
          },
          "user_tz": -60
        },
        "id": "ubrmficFA4X-"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "RL_Tutorial_Part2_Solutions.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3.8.9 64-bit",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.8.9"
    },
    "vscode": {
      "interpreter": {
        "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}