{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Building Neural Networks in TensorFlow\n",
"#### How to use TensorFlow's low-level API to implement a convolutional neural network for machine vision.\n",
"\n",
"\n",
"***\n",
"\n",
"Google's **TensorFlow** deep-learning API is a powerful tool that lets us take full advantage of the parallel processing capabilities offered by Graphical Processing Units (**GPUs**). With TensorFlow, we can train our neural networks faster, with greater control over our data processing pipeline.\n",
"\n",
"This tutorial introduces TensorFlow through its low-level API. While it's possible to jump straight to higher-level APIs like **Keras** which use TensorFlow as a backend, working first with the lower-level API gives us a better idea of what's going on under the hood, which is useful for debugging and customization later on.\n",
"\n",
"To demonstrate the use of TensorFlow, we'll implement a convolutional neural network (CNN) that allows a computer to recognize handwritten digits. CNNs are a popular and powerful approach to many image-based classification tasks. \n",
"\n",
"\n",
"### Dataset:\n",
"\n",
"We will be using the well-known MNIST dataset found here . The training set consists of handwritten digits from 250 different people: 50% were high school students and 50% were employees from the census bureau.\n",
"\n",
"\n",
"### GPU Specifications:\n",
"\n",
"The GPU used in this demonstration is the ***NVIDIA GeForce GTX 1060***: \n",
"* 1280 cores @ 1544 MHz base\n",
"* 6GB GDDR5 memory @ 8008 MHz, 192-bit bus\n",
"\n",
"The manufacturer was MSI. Seen in my computing rig below:\n",
"\n",
" \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Contents:\n",
"\n",
"1. [Importing the Image Dataset](#section_1)\n",
"2. [Data Visualization](#section_2)\n",
"3. [Building the Convolutional Neural Network using TensorFlow](#section_3)\n",
"4. [Training our CNN](#section_4)\n",
"5. [Testing on Unseen Data](#section_5)\n",
"6. [Testing on My Own Handwriting!](#section_6)\n",
"7. [Closing Remarks](#section_7)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Import Python Libraries:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import struct # methods for unpacking binary data\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"import tensorflow as tf\n",
"from PIL import Image # python image manipulation library\n",
"\n",
"# set Jupyter Notebook options\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***\n",
" \n",
"## 1. Importing the Image Dataset\n",
"\n",
"The MNIST image set is provided as binary data. We'll define a function below for unpacking it. \n",
"\n",
"* The 60,000 training images and labels are contained in the files **train-images.idx3-ubyte** and **train-labels.idx1-ubyte** respectively.\n",
"\n",
"* The 10,000 test images and labels are found in **t10k-images.idx3-ubyte** and **t10k-labels.idx1-ubyte**. "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# define a function for unpacking the datasets\n",
"def load_mnist(path, dataset_type='train'):\n",
" \n",
" labels_path = os.path.join(\n",
" path, '%s-labels.idx1-ubyte' % dataset_type) \n",
" with open(labels_path, 'rb') as lbpath:\n",
" magic, n = struct.unpack('>II', lbpath.read(8)) \n",
" # note: '>II' defines the byte sequence for unpacking\n",
" labels = np.fromfile(lbpath, dtype=np.uint8) \n",
" # note: reads data into a numpy array\n",
" \n",
" images_path = os.path.join(\n",
" path, '%s-images.idx3-ubyte' % dataset_type)\n",
" with open(images_path, 'rb') as imgpath:\n",
" magic, num, rows, cols = \\\n",
" struct.unpack('>IIII', imgpath.read(16)) \n",
" # note: 'magic' gives the file protocol\n",
" images = np.fromfile(\n",
" imgpath, dtype=np.uint8).reshape(len(labels), 784) \n",
"\n",
" return images, labels"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we specify the filepath, load the dataset, and print the dataset shapes to verify. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training Set --> Rows: 60000, Columns: 784\n",
"Test Set --> Rows: 10000, Columns: 784\n"
]
}
],
"source": [
"mnist_path = './data/mnist/'\n",
"\n",
"X_train, y_train = load_mnist(mnist_path, dataset_type='train') \n",
"X_test, y_test = load_mnist(mnist_path, dataset_type='t10k')\n",
"\n",
"print('Training Set --> Rows: %d, Columns: %d' % \\\n",
" (X_train.shape[0], X_train.shape[1]))\n",
"print('Test Set --> Rows: %d, Columns: %d' % \\\n",
" (X_test.shape[0], X_test.shape[1]))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***\n",
" \n",
"## 2. Data Visualization\n",
"\n",
"Let's print several instances of each digit to see some of the differences in writing style."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAyIAAAG/CAYAAACkKmPqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzs3Xm8VdP/x/FXSGQsZQyXREhI8qMiRUgaiTKlRKSUeUzKPFSGKJU0iUh9SchMSSlD5rmMDTcZMob7+8Pjs/ba9557Oufcc/eZ3s9/WtY+57Ta9hn2+nzWZ1UpKSlBREREREQkSutlegAiIiIiIlJ4dCMiIiIiIiKR042IiIiIiIhETjciIiIiIiISOd2IiIiIiIhI5HQjIiIiIiIikdONiIiIiIiIRE43IiIiIiIiEjndiIiIiIiISOQ2SObBtWrVKikqKqqkoeSOJUuWUFxcXCWZ5+jc/SeVcwc6f0bXXup07ipm0aJFxSUlJbWTeY7O33907VWMrr3U6dpLnX6vVEyi5y+pG5GioiIWLlyY+qjyROPGjZN+js7df1I5d6DzZ3TtpU7nrmKqVKmyNNnn6Pz9R9dexejaS52uvdTp90rFJHr+lJolIiIiIiKR042IiIiIiIhELqnULBERESlcxcXFrt20aVMA/v77bwA+//zzjIxJRHKXIiIiIiIiIhI5RUREREQkrmuvvRaAkSNHur6VK1cCcNppp2VkTCKS+xQRERERERGRyOlGREREREREIpfTqVlff/01AHfccYfrGzZsGAADBgwA4Pzzz3fHdtxxxwhHJyIiknt+/fVXAE444QTX98wzzwBQpUqwP9lBBx0EwIgRIyIcnYjkE0VEREREREQkcjkXEfn2229de//99wfgxx9/dH02WzN8+HAAxo8f747ZwjqJb/To0QD07t3b9f37778AfPzxx65v9913j3ZgWeTPP/8EYO3ata5vzpw5QHCNnn766e7YBhvk3FstaX5ZTyvnuWDBAgDat2/vjq23XnLzH2eccQYAo0aNcn3rr79+yuMsJB9++CEARxxxhOt7++23Aahdu3ZGxlTZ7LPK3qOx+N8LNvv/wQcfAMF3B8AVV1wBwN133+36Nt54YwBuv/12AM4555x0DDsr2Hv4oosuAmD27NllHjNu3DjXPvDAA4HgnIhkyl9//eXaRx99NBAuJ/3OO+8AsOWWW0Y7MFknRURERERERCRyOTNNu3TpUgBatGjh+lavXg2Ec1a32GILAKpVqwbAihUr3LEvvvgCgJ133tn1aWY18PzzzwNwwQUXALFnrv1zXSgs4mYzoAAvvPACAPPnzy/3eX70buDAgZU0usxZtmwZABMmTADgvvvuc8dsVvqrr74CwtdSstfQAw88AECNGjVc33XXXQcE7/NM+/TTT4HgM6lJkyaZHI5j12erVq0yPJKK++mnn1z7n3/+AYJZTn/m3t6v/vWYiKKiIgAuvPBC1zd27Fgg+F4BaN68OQAtW7ZM6vVzwc8//wzApEmTyn2MnSeA+vXrV/aQRPjll19Cf/o22WQTABYtWuT6XnrpJQD23Xdf16eoXfZSRERERERERCKnGxEREREREYlcVqZm+QuALSXLFh9Zyd7y7LfffgBcf/31ADRr1swdq1evHhAO2ffs2TMNI84Pn3zyCQB//PFHhkeSOX5BAysLbX/+/vvv7lhJSQkAu+yyi+vbaqutgCBE7C+utgWt+bRA+LLLLgPip3Gkk5XmhqCQQt26dSP5u9fF0ho/+ugjILOpWXZtQpAyZu/tXPTNN98AwWc7BClw6WBpg5aG5adw2PfD1ltv7fo23XRTIH/ey36RiWOOOQYIX0PG0vwaN24czcDyzIMPPgiEv1/fffddAO68884yj7diPAsXLoxgdJnx/fffu7adgyVLlpR5nH2n+ovPjaVM27mE4Pq133wQpAvnKztvlsr89NNPu2NvvPFGmcdPnjwZCLa1ePbZZ92x7t27A+E0zMqkiIiIiIiIiEQuKyMiF198sWv7ZRMT8fLLLwNBScaOHTu6Y4899hgAb731VkWHmDesZCXAoEGDQscaNWrk2rYY1BaG5QubnbLFz/fee6875i+OLW2fffYBgusNgpK122yzDQDLly8v81r5MosKcNxxxwGxIyLbb789EJQB9WejYhVBePXVVwGYPn162scZBZvNa926dYZHAmvWrHHtG2+8EQhv7Jpr16BFGu19BclFRPz/J/Za9l0AQcEDvxBKIZkyZYpr24zzKaecAoS/fzfbbLNoB5aDLPJo36u2CSTAmDFjgNjRplgFPBYvXgyEv4fffPPN9A02C8ydO9e1b7nllnIft9FGGwHhzzF7D/vFJYydzz59+ri+fFys7p+/Ll26AMHvDv8669SpExDOKLL3uPEfb5khUW1UqoiIiIiIiIhETjciIiIiIiISuaxKzbKwkZ/qUTqM6adade7cGQiHmGzhzZ577gnApZde6o49+uijMV+zEH322WcAtGnTxvX98MMPocfcdNNNru3X0c8nFtr0/63l2WuvvVz7lVdeAWDzzTd3fatWrUrz6LKbvRdLXzcQpF/Zwt51Ofvss4HgfWv7j/h69Ojh2v5eQNnA9rXIBraQ32fnNRdZSoUtwoTgs/zggw8Ggu8CnxUq+d///uf6NtxwQyDYAweCYhSFxham22cZwO677w7A0KFDAaVjGT/d8dRTTwWCPWx8ljJo+134vzUs9c9P543H0lnjpQjnqnvuuQeASy65pMwx28fMT8U899xzAahevbrrs5SsAw88EAinQm+77bYANG3aNJ3Dzji7Jmxh+rHHHuuO2TXaoUMHIEg3h2DRvv89Zd+nDz30UJm/55BDDknjqNdNEREREREREYlcVkREbAdqK1dnO+NCsOjo5JNPBmD06NHumC0I8/tOOukkILhztkWzEMzSTpw40fVZCVKLpBQKWzgXqxyyLWw6/PDDIx1TJvizrKXZ7KDtoGwloSEcCTFWarpQ2Psp1rlIli3C9EuJlrbTTju59gYbZP6j67vvvnNt+wzLBrEiVEceeWQGRpJeNvMJ0LBhQyCIcPgzq7bodciQIaHH+GzGFIIF/YXCysFaARJ/ofSZZ54JQNWqVaMfWBay3xg2ywzwxRdfJPx8P/Jm0WE/umJR9LZt2wKxS9f+3//9X+IDzhF2Dn777TfXt9tuuwFwzTXXALGj6f5nm8342zn2C+lY0Zls+J5IpxdffBGAo446qsyxE088EYD7778fCIpw+ObMmePapSMhfqleP/MoCoqIiIiIiIhI5DJ2u+jPfN58881AkF/p5wbahnG2IZw/u2UbXPkbXSXCvwu/9dZbgdgbCuWbWP9uv5Sqlba0mcRCYLmqlmtuG2dCcB0mWrJ4xYoVaR5dfvNnZyxP379GS/PLemcDm1GG+OOOipUs9zf2MvbezhelZ/tq1KhR5jH2md68eXPXF6tMaiHwN9GzzTdjqVWrFpB4lPORRx4BYkcJ/PWZuWrw4MFA/CiIlZYFmDBhAgAHHHAAELtUtl9G9q677gJiR0IsIu9nfOQLKzVr1w8EUfGBAwcC4XWbf/75JxCsH4Egs8XOsb/Wq3379pUx7Izwf5sOGDAACD7H7FxB8H6LFQkx/fv3L/fYww8/7Nr+WpwoKCIiIiIiIiKR042IiIiIiIhELvLULNt92nZchqBcr5WI9XcjtQVMa9eurZTxfPnll5XyutnEFv+vK1xpO6vXr1+/soeUNaw8pZUHrIgXXnihwq+Rr/wSoVZ28f3333d9f/31V7nPtdSaWDuyZ9J7771Xpi/ZNNF0uvLKK4HwIvrSi7rzlZ9ysGDBAgCmT58OhK+zBg0aRDuwLOGnpNn5sVKg/vvKT2MrzXZg91/LFhZbOXifFYL5+eefXV8ulAP239dPP/10uY+rW7cuALNmzSrTl6hYZcrNaaedBkSfJhOFOnXqANCqVSvXZ6lZtmN6165d3TErVvT555+XeS1Lr45VwjuXjRw5EgjSsSBIu7KiTJdffrk7VrrAhP3WhqDU9Keffur6rLS0pX41btw4bWNPVnZ9s4uIiIiISEGIPCJiMwD+poXm9ddfB4JFWj5/gZck59VXXwXgtddeK3PshBNOcO3u3btHNaScYpun+TN7Npvgzw4uWrQo9Dx/s6Fdd921MoeYERZpmzp1KhCeGSztiSeecO14C4a33HJLIFj0CcHGdLlQUvSggw6q1Ne3RZv+tXbfffcB4cWGxma7/AW1+ciP+Nj5sEXZfiTYyrD6G51Zqcp8XshuZWgh2ODRIiH+LH7pRep+WWo7n7FKnlukw/+cs1lY/zvGrtFs3iDXL9Pul9o19rlui6kTjYJYwQCLSAE8/vjjMV8b8mvBdWlWVtc+7322pYBftjjW962V7M6H0uTGLyphRYP8f7NFQqxEbyxW4tjK+UJQ9tdnmwj36tWrAiNOD0VEREREREQkcpFHRPr06QMEd7gQzEjFioSkU6ycWH8c+eaNN94A4PTTTy9z7LjjjgPCpQHzfdY0HluD5OfXW2m8WNG7WNeSsc0xx40b5/qybX1Dqr7//nvXbtGiBRA7bzdVdl22adMmba8ZJX8z1njsOrPr6OWXX3bHbN2arZuxEp8A//zzDxAuKd26dWsgeP/66+n23HPP5P4BeaBmzZpAsNbQL8k9fPjw0J8QzC5ajnmsjdRylUXQYpWftc+pfv36uT4r82zl9a20PgSfZ355fYt2WGltv4y1XXu5VtbcX29k71O/DK9FhJK9Th588EEAzjrrrDLHbLPOyZMnu758ug7LY2uAE3XKKae4tq01TMeGutnCPt8Bli9fXub4sGHDgKBUu2VrQBBtnDdvHhDO4LCoSqzNS7Nh/WB+/DoSEREREZGcohsRERERERGJXCSpWW+99ZZrWxlPP0TkL2arTJYe4//dmSxZVhn81BB/sVdpFhJNdNfwfOKHP7/55hsgSDOyhXIQlE20FIZjjjnGHbNSlrEWM1rZvCeffNL1devWDYD111+/wuPPFpbWmEh6o6UgQfw0NVukfv7557u+TJbEjccvq2mfKe3atQNgjz32iPtcC5/bubPFmxCkZNjCd7/UuZVX9c+JvYftOrWwPcTe2blQNGnSBAiX77VSmP6Ozj169ACCFENLM4LcKDcbz0cffQSEF64aK6/bu3dv12fXjl1zflqqLTD3U4uuuuoqIEjl8v8ee7y9J/y+bOYXnPBTJlNhJWkBzjvvvDLHrQCH/b8ohHQsCL4Pnn32WdcX73vk1FNPBWD8+PGVO7AM838fbLvttgAsW7bM9VnaabzCGjvttBMQLgRgv2v8tMpGjRqlYcTpoYiIiIiIiIhELpKIiF+SzBbPbb/99q7PL1mXLjYrbeUrfccff7xrX3HFFWn/uzPp9ttvd+14M8+XXnppFMPJKhYJefvtt11f6XKrtjkSBJstWXnG33//3R1bvHgxAPPnzy/z99gMxhlnnOH6rKyl//f5s+C5YrvttnNtK4Zgs8u2aBoSWwA3duxY17aN0XLJ4MGDXduukZdeeimh59arVw8IImX+os1ddtklqXFY2WS77gppQ9JE+NesLTT2owBHHHEEEJRt/fjjj92xWCWRc4n/WVeafw6MZSfMnj27zLFY5fVtEXysQjN2Pgvxu8bYInSIPYs9bdo0IHeLc6TqnHPOAWDMmDGuL94sfz6X1vb5BYPmzJkDhDNbVq5cCcBee+0FBJEiCDbAtAi5f8wiInbes40iIiIiIiIiEjndiIiIiIiISOQylhvih6DSuUDLUrLuvfdeINh9E6CoqAiAK6+80vVlQw3ldLAdcP260qX5qUKFsojVX5h+xx13AOFrwliKjIU3IbhGrTZ+27Zt3TFLU6hWrZrru/XWW4EgHcLfR+Swww4DoEuXLq7P9imJdf3XqVNnHf+yzLOFp1aPPFlWBx5yMzXLZ3v1xNqzp7LNnDkz9N+2+FrKsve0FaeAYIGofXfMmDHDHbM0rXUVH8hWq1atAsILgf3vAQjvnm4L++3xtvcFBOlXtjAdggIesR4fa4F8obD9HtZVpMNP3cpXv/zyi2tbqqPtX+anXNl3pJ2T2267zR3z9/cqFPZ71V+snohPP/0UCH+O2bWXrWm7ioiIiIiIiEjkMhYR8RfSVJQ/o2M7wdqiY3/2x99FPN9YGWJ/tsocddRRANx9992RjimTbCbK30HZFk36JTlt8aqdIz9St3TpUgB69eoFBKWnAfbZZx8AHnroIddnsw1WkKFv377umO3e7JcfnDp1amjMtqAd4JNPPlnXPzHn+aUtJX06deqU6SFkFX829bHHHgOC8skQREKMP0sdaxF2LvJnnuMt/C1d4n7hwoXu2OWXXw6Ei3Y0aNAg9Dg/QlyILAJv58OPgtg59bMWatWqFeHoMmPRokWuffbZZ4eO+b/JTj75ZCB4b/oRkX333bcyh5hXrDhUrGvP34IgmygiIiIiIiIikYskIuLnp1rbZqIBrr766pRe1zaV82eeV69eDUC/fv2AIFcz361YsQKInYdqkYB8WQ+TCMub90tH2lqMJ554wvUdcMABQJALPnLkSHfMNvOyGUA/omRrSjbffPMyf7fNCjZs2ND1WWSmc+fOrq90hC7brlV/fc27774LwN577+36bDOuZNkmVlFtZCqFxUpcjhgxAgiv1bINTGOxtSKWmw25Xza0Q4cOQHhdnJ0Pi3D4Gz7+9NNPoef7n0n23e1vimbr4nJ948eKWLt2rWvbZ1usss+2oeHRRx/t+nL9+orHvlP97zxjURLLLIBgc+A+ffqUebyVR5d1889prlBEREREREREIqcbERERERERiVwkqVmxFsr5IXLbobhnz55AOMxrYeNRo0YB8Oqrr7pjS5YsAcJhu5NOOgkIUrPy3UUXXQSEywSW5qcIFYpzzz23TJ8tSvXLN1sqwnvvvVfua1kpaLs+If6u9fE0b948ZjubWPm/QYMGuT5LNfjhhx9cXyKpWZbWtmDBAtdn71ELxfuqV68OhIsGSGIsdcaKLEC4AEI+smvIT7e075NECz60bNkSgJtuugkI0jXzgb1H/RLhds7q1asHJJ4eZOW6zzrrLNe33377pWWcuciKklxwwQWuz36nGD9Fy1KU8jkdy/fUU08BQbo8QMeOHQHYf//9gXD67wsvvAAE3zF+Sv92221XuYPNI5ZGnUsUERERERERkchlrHyvfydsM1hjx44FoGbNmu5YvLs7K0XmL/6yBWH5zC9XbKUAbYbeL59om8RtsskmEY4uO8TaDMjK2s2dO7fM40855RQAjjzySNdn19eWW24JpB4FyTXdu3cHYP78+WWO+YtXYy3UL81mql9++WXXF2tG0ErO2iaH2brxUjaz8xovOprLfv31V9f++uuvgeB9+9ZbbyX0Gq1btwbg2muvdX1WrjcfZ6p33HFHAF566SXXd/311wNBKeNYbJbfjw7ZLHa+lDSuKIuml46CAOy1114AHH/88ZGOKZuULgXtt+33nx8pt+IlVtLYLzTTvn37yh1sHvniiy8yPYSkFcYvKxERERERySq6ERERERERkchFkprl7z1wxBFHAPDcc8+VeZwtYPdTj8zWW28NwDnnnOP6Ut1/JNf5i3xLnyu/Br4f2iw0zz//PBDeQdlSsvyFbyeeeCIQLI62vQQktiFDhlT4NbbffnsATj31VNdnqTIbbJCxbNG8YYs+AVq1apXBkaTO3727f//+AMyZM8f1ffTRR+t8jTZt2gAwcOBA12eLq1PdAydX+YvKH3nkkQyOJPfZPjVDhw4tc8wKw7z44ouRjikbLV++vEyf/Y6zlLXHH3+8zGNskXujRo0qcXT5q0mTJkA4RTfb08qze3QiIiIiIpKXIpl+9Be12uLqCRMmuL54pXavu+46AHr16gXAVlttVRlDlDxji/ZbtGjh+vy2lM9KTt55552uL9bsXzy2WNPe+7ZIGIL3skoyppdf7jLXWCn2G264AQhHzP1yxOWxss8QRO2shPeGG26YrmGKuOvrnnvuKXPMCsRYqeNCFmvbAFvYb59VtWvXdscscpmLO4NnE/tebdCggev78MMPgXCUapdddol2YHEoIiIiIiIiIpGLPCHbNlbyN5yLtfmclG+HHXZw7WOPPRYIb+glUhF16tQBgtlpgEMPPRSAM8880/UVFxcD0KNHDwDatWvnjln0yd9ITSqHbZQ2cuTIDI8kddOmTQOCEu6x+DnjXbt2BYI1Rf4me9oMU9LNLwNvZXvNFVdc4dqHHHJIZGPKdlZyd9y4ca7PtlewMvlWsheCjW4lPYYPH+7aRx11FACXXHKJ67v77rsB2GabbaIdWAyKiIiIiIiISOR0IyIiIiIiIpFTrcwc5Ke7zJgxI4MjkXzml9Jt27YtEE5RkOxgJXpzeUf1Cy+8MPSnSDaZNGmSa0+ePBmAevXqAdC3b193zF98XegsRfK0005zfX5bKlezZs1cu0uXLgBMnTrV9dkO9nfccQeQ2aIeioiIiIiIiEjkFBERERERKYcVhQG47LLLAJg4cSKgKIhkJ9vCAIKCAXvssYfrszLUgwYNAjK7aF0RERERERERiZxuREREREREJHJKzRIREREpx5577unaf//9dwZHIpI8S9O65pprXJ/fzjRFREREREREJHJVSkpKEn9wlSorgaWVN5ycsXNJSUlSK9R07pykzx3o/Hl07aVO565idP5Sp3NXMTp/qdO5S51+r1RMQucvqRsRERERERGRdFBqloiIiIiIRE43IiIiIiIiEjndiIiIiIiISOR0IyIiIiIiIpHTjYiIiIiIiERONyIiIiIiIhI53YiIiIiIiEjkdCMiIiIiIiKR042IiIiIiIhETjciIiIiIiISOd2IiIiIiIhI5HQjIiIiIiIikdONiIiIiIiIRE43IiIiIiIiEjndiIiIiIiISOR0IyIiIiIiIpHTjYiIiIiIiERONyIiIiIiIhI53YiIiIiIiEjkdCMiIiIiIiKR042IiIiIiIhETjciIiIiIiISuQ2SeXCtWrVKioqKKmkouWPJkiUUFxdXSeY5Onf/SeXcgc6f0bWXOp27ilm0aFFxSUlJ7WSeo/P3H117FaNrL3W69lKn3ysVk+j5S+pGpKioiIULF6Y+qjzRuHHjpJ+jc/efVM4d6PwZXXup07mrmCpVqixN9jk6f//RtVcxuvZSp2svdfq9UjGJnj+lZomIiIiISOR0IyIiIiIiIpHTjYiIiIiIiERONyIiIiIiIhK5pBari4iISOFavXq1a1955ZUAjBkzBoBvv/3WHatdO6kiVyJSoBQRERERERGRyOlGREREREREIpd3qVkff/wxAHvttRcA//77b5lju+++e/QDy4AhQ4YAMHDgQACaNGnijs2ePRuALbbYIvqBSV5avny5az/zzDMA3HTTTQC0bNnSHfOvQ3PyyScDsP7661fmEAU44YQTACgpKQHg0UcfzeRwJEd88cUXADRq1Mj1bbfddgBcc801AGy22WbRD0xEcpoiIiIiIiIiErm8iIg8//zzrj148GAA1luv7D1Wr169ADjvvPNcX+fOnct9fC768ccfXfvOO+8Egn/bokWL3LGvvvoKgH322SfC0WW/l156ybWnT58OwOuvvw7AW2+9VebxxxxzjGs//PDDAGy00UaVOMLsM3PmTAC6devm+n755ZfQYz788EPXHjFiRJnXsChJ/fr1K2OIBe/666937SeffBKAAQMGZGo4kiPefvtt1z7kkEOAYIE6wCWXXAJA1apVox2Y5C2LrA8fPtz1WRGEiRMnur4zzjgDgBYtWpR5jS5dugCF912cq/Lj17eIiIiIiOSUnI6IWCRk5MiRru+1114r9/F2zH9McXExkD9rJapXr+7a7dq1A+CBBx7I0Giy32+//QYEUbLx48e7YzVr1gSCqMcee+zhjs2YMQOAJ554wvXZjOGbb75ZiSPOPq1atQJg0003dX2lIyLr0rRpUwBefvllABo0aJCm0RW222+/HQhHRDbccEMAjj322IyMSbLfqlWrAGjWrJnr69ChAwBXXHGF66tSpUq0A5O8d//99wNwyy23lDnmZ67Yd7X/nW0uvvhiAMaOHev62rZtm9ZxSvooIiIiIiIiIpHTjYiIiIiIiEQu61Oz/vjjDwCWLl0KwHHHHeeOff/996HH+KzE4D///OP63nnnnUobZ7awtAuAXXbZJYMjyQ2WnvLBBx8AcOutt7pj55xzDgAbb7xxmeetXLkSgHr16rm+xYsXA0GqYO/evSthxNnHzs+oUaNcX9euXQH49ddfAdh1113dMSsD6vvhhx+AINVNqVnpMWfOHAD++usv12efoZZKKOX76aefgPD5mzp1KgDXXXddmcdbGerbbrstgtGl399//w3AmWeeCYSvkXHjxgFKx5LKdd9995V77NBDD3XtoqKich83bdo0ILhmoXBSs37//XcAnn32Wdd3/vnnA0GRIp+d7549e0YwutgUERERERERkchlZUTkjTfecG0rQfvQQw8B4Q0K45XctY3U/McfffTRaR1nNvKjQ7HKzQq8//77rv3KK68AQfTjggsuSOg1ateuDYRnRfv16wfA0KFDgcKJiBg/WrnvvvsCQWGIWrVquWOxIiKm0M5ZLJ9++ikQbERqizchdnSutFdffdW17fzbBq8Aw4YNS8s4841FRe27BoJS06tXr3Z98SICfin5XGSfXbYh6bJly9yxatWqZWRM+cb/TeJfV6VZAZ0NNsjKn2mRsm0GrFQ8wCabbBJ6jEXfISjDb8VPIHhvHnzwwUC4sE8+sO/Vc889FwhHROwzK9Zn19lnnw2ES+xHHdFVRERERERERCKnGxEREREREYlcVsX8LIzWsmXLch/jhzXjKSkpSfm5uWzt2rWubakGsdhu4TvttJPry5e9VNbFP0d77703AN27d0/ptTp16uTalpple5P8+eef7lihpTXY/hUXXXQRAHPnzk3oef7/m0J1wgknAPDuu+8CMGTIEHdst912W+fz/fTCFStWAOH9brbffvu0jDOXXXrppa5t+/7ES6vyPxv79u0LQPPmzQE4/PDD3bFcTKPulcJMAAAgAElEQVSxBeoQpEJ37NgRgM033zwjY0qnRYsWufYBBxxQqX+XfeZbWvTdd9/tjlnBA7/wweOPP17ua1nxj169eqV9nLnG3n+l07EgSEd/8MEHXV+sRdmtW7cG4KyzzgLg3nvvTfs4o2a/NSAoEGPX+7bbbuuO2S709t0CMHr0aCAoruN//lmRp/XXX78yhl2GIiIiIiIiIhK5rJi+sUhIly5dgPAidFucWadOHQB+/PFHd8xKqPrs8XbnvGbNGncs3uL2fLHZZpu59oABA4BgIbbP+rbaaivX58/u5zO/NKzNHlStWjWl14r1vO+++w6A2bNnuz5/IXch+L//+z8Ann76aQCOOOIId2z+/PnlPu+qq64C4pdwzHc2C20LC/0Z1Hi+/fZbIFjsDsFnnh+dKzRWzhJg8ODBQLhMtxWeaNGiBQA33nijO2Zlp/2y6PkWOfaLIfz8889AUOwlHzRs2DCtr2eLou27w8q2Azz33HNA7EjHYYcdBgQz1/5rxYrG9e/fH1BEBOCTTz4Bwp9t9puwXbt2ALzwwgtxX8M+C/3zn+uOP/5417br8aSTTgJg0qRJcZ977bXXAvDoo48C8Pnnn7tjy5cvB6KLnuf/L3MREREREck6GYuI+CV6bU1IrIiFldy1TaT8mYNY5XhtA5uDDjqozOMLjeVCxoqIFLJ05nHXrFnTtS3/2GYm/DLBhRYRsbLIFv1YsGBBQs9r1apVpY0pm911112uPW/ePAD2339/IP7GXRBETGwm348CH3XUUUBhb15o65UAbrnlFiCYDYRgvYgf9Sgk/voh2/Rtxx13zNRw0i7VaHd5bG3Q22+/DYTXo1oU0/psY0iA4cOHA+F1DrbpsP1O8V/LMhokWOtWv379pJ63++67u7ZFQf1NEXPdpptuWqbPIiLJqlGjhmv7mTVRUEREREREREQipxsRERERERGJXOSpWRaC9MuIGVto7qdc+SkLpTVr1gyA8847z/VZ2UFjYVSAI488EgjvOFkIrGxxISzWj5pf3q7QSvQaKxph5REB3nvvPSBcGjQR/msUgliLgy2VZPLkycC6dwC2NCMrw+iX5J41a1b6BpvF/LLPVujAStH6ZT3tu2W//fZzfblYcjcdbOGvFZSAoOBBPB9//LFr28J9v1RovrPvUXtf+r8xLD3SUq788seWtjVx4kTXVzpt2k89uuyyy9I57LxnBVIAnnnmGSD4TQnRlaKNkp/KZ21LF/e/e4uLiwGYMGGC67N06Z133hmAGTNmuGNKzRIRERERkbwX+VSQRS9++eWXMsdsMVePHj3Kfb4t4ASYOXMmEHuTG+MvQFzXzGK+shkcm5GR9PFnHawUo8mHzcAS8eWXXwLw0Ucfub5kIyHGZrGvueaaig8si33//fdAUNbYyiVCEOHwF1qWZtESgNtuuy10zM5hIfE3jrNNNG22ed9993XHCjX6Ecv48eOB8CZ/fvENCBd7sUWwq1atcn0bbbQREESdOnToUDmDzSJW7tjKx5Y+Z+WxIia9e/d2fVZaul69ekBQqAJiL0TOZxZZ9zfpi8d+1zz11FNAkCEDwXWZ71577TXXtt93gwYNAsLRkhdffLHMc+fMmQOEI0mZooiIiIiIiIhELpLpIT/v1DYktHULEGwnn4hEZx9isTtE/+8WqYjVq1e79jvvvBM6dswxx8R9rs38LF26FAhmKCCYWbTN1rJZkyZNgHDu82mnnQaEN5NLRCI56rnGPm/8WSlbCxNr/ZZt8Gp596effro79scffwDwwAMPuD77XLNyn1aCtZBccMEFrm0zg2eccQagKEh5rJSxldqG4FzZd7K/hsHK/PpZCQsXLgSC69mf0U/3RoLZIpl/l/+Z2KdPHyD8mWhrSWxme8stt0zHELOefe59+OGHrq99+/ZAsJ7BZ2uR/NK0tvltVJvuZaPtttvOtX/66Scg2NgxVllpP8pm1142UEREREREREQipxsRERERERGJXKXGrJctWwYECzIhCLtFVUrWdh2GICSqMraSCn8BtpVdtd11YznllFNcu2nTpkB4cZktWLbF3hZ+hmDht78rdLY7/vjjXdsWX9p58lnah19q21I289Grr74KBLucQxAqt8+ivffe2x2z0Lr9+fDDD7tjVnL166+/dn2WwmU7Bxci/zvGzpuViPd3DvfPc6Gy72X7PItV1vSrr74C4MQTT3R9sRa12ufaJZdcAsDAgQPdMb8caKF58803gfDO6lZi2i/RO2XKFAC22mqrCEeXOfY5P27cOCAoLLEulorarl27ShlXrrLUSAh+R1jhE3tv+nr27Ona22yzTSWPLnH6RS4iIiIiIpGr1IiIler97LPPKvOvictmI6HwNjI08TY09M9Jp06dIhtTJvmbn9mCcYtA+NdL6c3g/JLTVooxHtswCILShD6bDTr22GOBcLnfXF+06JdMLc0W0V133XWuzz4rbMG+LbyDcKQoV8ydO9e1bbbeLyVuRTeee+45ILyBVP/+/QGYPn06EMzwQ3Du/FLcNgNmiw/9a7MixT2yzZIlS1x7xx13BILZ/Mcff9wds9nWvn37AuGynrYZ39Zbb12pY81mfoENCDY089kC4CuuuCKh17SStIW8cBiCCHnLli2B8HfNrrvuCoS/YwohEuJHb1u0aAGE38uJ2G233dI4ovxkn//xShdffvnlUQ0nKYqIiIiIiIhI5HQjIiIiIiIikctYgXW/vnZlsFQYf8Gw8XcsLoQa8/F2Vh89erRr246c2bSIqaL80Pgdd9wBBLsAQ/zF5sbSpPzUKbtuYu0gfumllwJw7rnnuj5LJZFgsbqlY/mqVasGxL5Wc8mwYcNc29IK/B3PjzzyyHKfa7uEW3GNp59+Ou7fZelatvdMPqRjrVmzxrUtddHSqiBYwH/YYYcBsPHGG7tj3bt3B4LULL9ggr1uIadmlRYr9dHeh4nK9VTSivC/Qw488EAg+Izbc8893TFL1yyUc2V7Qh166KGuz4ogGL9Qgu09ZamVkhpbCuHvl5ftBZqye3QiIiIiIpKXMhYOqKxZd4uEtGrVCoAVK1a4Y7YLpb9Ae5NNNqmUcWQT24H0+uuvj/s4i47Y4/OBzbJAMIvqL+Y69dRTAahbty4QlPwEqFq1KhDMnvoRkf322w+AxYsXuz4ry3jttdcC4cXJEhg6dGi5x2wBv3+uc5Ff9tTK9ib6b7IZfH+XamMLXe169eXTTKtf4tRKfk6YMMH1WSQkljFjxoT+u0uXLq69ww47pGuIOcsiaP7OyxX1/vvvA7lZWCJVVqLXFqZDEAkxL730kmvn0/szEbZTeukoCASZKldeeaXre+eddwBFRCrKft/4URArRJStGUCKiIiIiIiISOQq9fbIZlz8XDXjb0BVehYhUbZZoZ+LX/pu2p9Zs9mJ2rVrp/T35aqGDRtmeggZ89BDD7m2rQ16/vnnXV+dOnXW+Rp2/d52222u7/PPPweCKBvAM888A+RPJMTWKACcc845APTo0cP1+bm/6+Ln/N94443lPq5NmzbJDDFr+ZG1RPzxxx+uPXnyZCCIBOy1117u2CGHHJKG0WW/wYMHu3a/fv2A8IaZpTVo0MC133vvPSBYm3PLLbe4Y8mufchHtv4qHeuw7Lvb1kSdf/75FX7NbGcRyz59+oT+G4Lvg8ceewyAWrVqRTy6zHr55Zdd2yIcfsRo5syZQDAz768Rueuuu6IYYt6y7B9bY2gb3QJceOGFQPZG5RQRERERERGRyOlGREREREREIlepqVm2c7Ltlgzwww8/lHlc586dgSBU7C/03GOPPYAgVO8vsLPUkViLz4cPHw7Acccd544VWkqWsfPrlxL84IMPyjzu6quvBoJUt3woA+qnH1iBhER3/7XSvD179gTCJadtQZi/63W+lei1MsQA48ePB8KlKqdOnQoE6Qf+9WK76doOuv6OrpZy5LvpppuA8A7jhcQvKW2fm5bm4e/SXij8FEBLp5o/f77re/TRR0OPtyIlECyEvf3224HC2L06GfY5ZX9aSilAu3bt1vl8P5Xadl7/8MMPARg1alTaxplNfvvtN9e2QiX22WY7pgPMmjULCG8RUAjs/XfWWWe5Pktp9hdIx0uNLF1kwk97ztaUokzzU3oPPvhgAJYuXQqEi3v83//9X7QDS5IiIiIiIiIiErlKjYjYDLyVnARo3rw5EI6MzJgxAwjKjf3vf/8r9zVjbdLSsWNH12eLag8//PAKjT0fNWnSxLVtBsuX7ZvepMJfqG+ROX+mv7i4GIADDjgACEeN7HEWBWjdurU7ZrM3iSx2z1X9+/d37U8//RQIb65n0cp69eoBcNBBB7ljTzzxBAA//fRTmde1KJXNLAIMGDAAyN7ygpXFzs+tt97q+uz8WBQp10sZV9TJJ58c+hPCm0NKcjbddFMg2OC1a9eu7tiUKVOAoOS0H2n65JNPgPDngmUgvPjiiwBUr169soadUd98841rWyTEPqtmz57tjvnRkUJiEaPvv/++zDErHeuz791Jkya5Pn/zYQhHRRPNYig0Q4YMcW2LhFgRhW7dumVkTKnIv1+eIiIiIiKS9XQjIiIiIiIikYskD8JfuPXZZ58B4YU0Vic+EX4qTNu2bYFgUSKEd82WMP882+LjfOcvrrY65X5qli289BcLm+7duwNw//33A4W3H4ufZmA7WVvqIwQ751ralv25LrZ42HYmLmTNmjUDwufO9mKwELtIZejQoQMQ7FsDQZqWvwjW2K7p/h4v9nng7weRjyydDYJ0NCuqU6jpWD7bP+TXX38tc8wKGkCQHj1v3jwg2I/LZ6n2/q7rEmbFhiy9EoK0yFNPPTUjY6oIRURERERERCRyka8MtfKc/myfLRAuvTgYoFGjRkBQ3nOnnXZyx+rWrVu5g80zRUVFrm2LsxctWpSh0USvb9++oT8lcZdddhkQlDSGcFQTYMGCBa5tu7uaGjVquLYiIQFb+Hv22We7vi5dumRqOFKALDICsWe0JbxY2hauK/siYIVcdthhB9f37bffAuHCRP7i9NJsRt8iIfFK/RYqK31vGQo+28aicePGkY4pHRQRERERERGRyGVFrcyWLVsC8MYbb2R4JPnNcnwhvDmYSKL88rqlc1H9/7b1OBKfbZZpf4pIdvO/R+U/2267LRD+XXHggQcCsUv6Gr909FVXXQVA/fr1K2OIOcsva2wbda9evRoIbyDpl8/PNYqIiIiIiIhI5HQjIiIiIiIikcuK1CwRERERyV3bbbeda/u70UvqZs2a5dq2k/oxxxwDwJ133pmRMaWbIiIiIiIiIhI5RURERERERLLE0qVLgWCDWwgiIFYYxi8ek8sUERERERERkcjpRkRERERERCKXH3EdEREREZE8sPPOOwOwZMmSzA4kAoqIiIiIiIhI5KqUlJQk/uAqVVYCSytvODlj55KSktrJPEHnzkn63IHOn0fXXup07ipG5y91OncVo/OXOp271On3SsUkdP6SuhERERERERFJB6VmiYiIiIhI5HQjIiIiIiIikdONiIiIiIiIRE43IiIiIiIiEjndiIiIiIiISOR0IyIiIiIiIpHTjYiIiIiIiERONyIiIiIiIhI53YiIiIiIiEjkdCMiIiIiIiKR042IiIiIiIhETjciIiIiIiISOd2IiIiIiIhI5HQjIiIiIiIikdONiIiIiIiIRE43IiIiIiIiEjndiIiIiIiISOR0IyIiIiIiIpHTjYiIiIiIiERONyIiIiIiIhI53YiIiIiIiEjkdCMiIiIiIiKR2yCZB9eqVaukqKiokoaSO5YsWUJxcXGVZJ6jc/efVM4d6PwZXXup07mrmEWLFhWXlJTUTuY5On//0bVXMbr2UqdrL3X6vVIxiZ6/pG5EioqKWLhwYeqjyhONGzdO+jk6d/9J5dyBzp/RtZc6nbuKqVKlytJkn6Pz9x9dexWjay91uvZSp98rFZPo+VNqloiIiIiIRE43IiIiIiIiErmkUrNERESk8Nx5550AnH/++a5v1113BeDDDz8EYMMNN4x+YCKS0xQRERERERGRyCkiIiIiIs6///4LwMiRI13f5ZdfDkC3bt1cX9OmTQEoLi4GYPvtt49qiCKSJxQRERERERGRyOlGREREREREIqfULJFyzJ8/H4A///wTgFdffdUdu+qqqwDo3Lmz6+vXrx8Au+yyCwA77rhjJOOUwmZpMb179wbggQcecMc23XTTTAxJctycOXMAOO+881zf7bffDsCAAQMyMiaReCZPnuza9ll43333ub6uXbtGPiZJjCIiIiIiIiISuZyLiCxdGmywWlRUBMB665V/P+UvtuvVq1eljSsb/fPPPwB8/vnnAPTv398dmzVrVkbGlG3Wrl0LwPLlywG44oor3LFp06YB8Mcff5R5nl1z06dPd33W3mOPPQBo3769O3bNNdcA4fKW8a7bqPz111+u/eKLLwKw8cYbAzB37lx37KeffgLgrrvucn0dO3YEoE6dOuv8e3bYYQfXtvOy0047pTrsrGCRMgiuo2rVqgFQtWrVyMbx/PPPAzBjxgwAHnzwQXfszDPPBLLjWpPsZ9+vXbp0AeCwww5zx/r27ZuRMUlhWrZsmWtb8QSffU/VqFGjzLHffvsNgHvvvdf1KSKSvfTtJCIiIiIikcu5iMjVV1/t2jbLF2+279xzz3Vtu8M+6aSTXF+9evXSPcSsYTO29evXB8Iz12vWrAEKK4f8u+++A2DBggWuz2aRJ06cmLa/5+OPPwbglltucX3WHjFihOuzyMB2222Xtr87WXfccYdrX3rppUk91595T4blmDdu3Nj1WbTS1txsueWWKb12lEaNGuXaF1xwAQCTJk0Cwp8xla1Ro0ah//Y/844//ngAatasGdl48snPP/8MwD333OP6Xn75ZQCefvppAI4++mh37KmnnopwdOl39913A8G6o9NPP90d22CDnPu5IDnk999/B+Cyyy4DYMyYMe6YZSVUqVLF9dWuXRsINtOMZffdd0/7OHNRrIyGwYMHA/D3338D0KdPH3fMf1wUFBEREREREZHI6UZEREREREQil/Wx1h9//BGAQw89FICvv/465dcaNGgQAPvss4/ry+fUrNK++eYb17ZQXSGlZllKll9yN566desCiackfPXVV0AQYo7FD39aSpa/qD1q48aNS+hxW2+9NQDNmzdP6PF77rknEITNV6xY4Y5ZGWQ/Rc7aBxxwAAD77bdfQn9PtrG0qN122831+SlolWHVqlWV+vr57rXXXgPglVdeAYLUKwjSr+Kxoim56pNPPnHt4cOHA8F13L1790wMKSMsRcUWOvsWL17s2iUlJQC8//77Sb2+FSqxdEkIvn8LtZjEZ5995tpWKOaxxx5L6LkrV64Egu9bP53Ivq8s1bAQPfvss65txSf871UrWGT/D/zfJgMHDgSC9LfKVphXv4iIiIiIZFTWR0SsBG28BUmSGJvJKXQ9evRw7fvvvx8IFvJfcskl7pgtoPZL7sbz6KOPAnDiiSemZZxRsI3LIIjoxCqra+cg1QiaX+p27733BuCLL74o87hHHnkEyN2IiC1u9hcwv/nmm0B6yxX7ZZevvfbach9nJaV79uyZtr8719j/EwgWnfsbnX355ZfrfA3//6cVTDnkkEPSNcSM+uCDD1zbvm8LcTPW888/HwiXfK0MVlIbgoiTFTGxkrT57u233wbCEfbSmQR+YSL77eJHKK1suUU97HMWYNtttwVgo402Suews5q9d8ePHw+ENyO94YYbgGCjRwjOjW034EdEoj5vioiIiIiIiEjkdCMiIiIiIiKRy/rULFtgnigL3dmC2Ouvvz7dQ8pZfg1uP1WmUFh6xTHHHOP6LGRpO2FXZP+K0vs5xLLZZpu5ti2oyyR/f4nK3Gti/vz5rh0rJctCwWeddValjSHdLKQdixXZALjyyisBGD16tOuraOjbX/zvL0qUsvz9cUaOHFnmuKUrnHrqqWWO5Uv6VTyWUgpBSlYhLVI3tleFv5t8OlOlbDH8c8895/oeeOABAPr16wfkbkpqsvbff38gvEi/WbNmQLAXT/Xq1d0x25vL//9x8cUXA8FeYJaaBLmVHp0udt4spXzq1KnuWLwCPZbi5u9n5v9OiYIiIiIiIiIiErmsioi88847QGIzyxDsCu0vyjGrV68G4N9//3V91vb7CpUtFtt1110zPJLoxJqFTrU8nc2+3H777a7PXwBbngkTJrj2wQcfnNLfnQvs/FxzzTUADBs2LO7jrYSoFQ3IBUceeaRr2yJof1dzM2XKFABOPvlk1+cvfk7FFlts4dqlSyX7OnbsWKG/J5fddNNNADzzzDOuz6IfNpsKhfUZ6LMIwIsvvuj67LxEVbYzm1i0zJ+lX3/99dP2+va7o1OnTq7v8ccfB2D27NlA4URE7Bz7WRoWfbTsBJ9Fn4877jjX17Bhw9Bj/Ijmrbfemr7BZjF/gb9FQs4++2xg3Z/9toXDLbfcUkmjS5wiIiIiIiIiErmsioiYRDf3iRUJMXanHeu1CmXzIPt31qhRAwiiRKByyKn46KOPXNtmWydOnJjQc+vXrw9Ay5Yt0z+wLOGfn7FjxwLhiJGxGa9p06a5Piu3mEv8z5HTTjsNCCIj7733XpnHX3fdda5teeip5qDbbBbovVyabVB4+eWXA+Ho08033wzA5ptvHv3AsoxFAL7//nvXV6jRIYg9E59OVnLboiC+QlvTYO/Dyy67zPVZFOPdd98FYNSoUe6Yle9t06ZNmdeydU2FtB7Yomv+GjbLJrDv3HX9zj3llFOA4HvbftNkQmH8IhcRERERkayiGxEREREREYlcVqVmXXDBBet8zLp2J7ZFssuWLUvLmHKZLc62BV7+QmlJnJVb9MPCfqnA8tiOrwDt27cHUt+ZPJstXboUgH322cf1xTs/FjL2F6b7ixZzkb3XWrduDcROzXr99ddd+4cffgBghx12KPc17RxaeUqf3svlGzJkSOi//XKs9v/FTwUs1HQkKx3rO/zww6MfiBScAQMGAPD++++7Pktzti0YmjRp4o5ZapZfttxSsl555RUg/mdpvrH008WLF7s+S9GNl+5r5wqC82xpqv7O6lFTRERERERERCKXVRGRpk2bAuG7ttJizQ76Hn74YQD69++fvoFJwfjuu+9ce+bMmUBQFGFdURCbibAyrV27dnXHKrJRYrZ76KGHgMSiRBBspumX6baZWFu06Zdp9DdaynZWjGDo0KFxH7do0SIgmMXzN3mcN28eECxIt83OEmWbhUF6N2TLZuecc45r20yfLVLfeeed3TFboHnUUUe5vkJbwG4lP+368t9rm2yySYVe2y+i4JeYlmAWW4KyyPfee6/r23vvvYFgAfvy5cvdMYuI+N8Fc+bMAQorEmLsvO27776ur27duuU+fs2aNQCcdNJJrs8WvNum4RV971eEIiIiIiIiIhK5rIqIWPm1WGXHunfvDsS/64PMliDLJcXFxZkeQlawWfxVq1YB4VKffv4qhDe32nDDDcu8lm1o2K1bt7SPM5udcMIJQHhdhK2r8We14rFN1exPf2NA+1zwy3VncvYmHrt+/Hxbf62QSWSjQZuxSrbc+JtvvunaFl32IwD5wCJIdp4tCgLw4IMPAnDssccC4UiHrWey0r4QREQKhZXrtXPhlz1N5Fqz2VUIylXbmicrCQzBuqmBAwe6vlgbfhaKlStXlumz9UnbbLNNmWO26bBFkNfloIMOqsDoMsPfZPjQQw8FguiHzz4Li4qKXF8hRkKMbZLrlzguvfmmf9106dIFCH8fX3XVVUA4mpwpioiIiIiIiEjkdCMiIiIiIiKRy4rULFtAaOE3n5UEHT16dEKvZWG9WK9li4c7dOiQ0jjzyfjx4wEYNmxYhkeSWffffz8AvXv3Lvcx7dq1A+CMM84o0ydBeoG/y7wtWv3555+BIHUDgrCy7aQLZcPx/vvX0mgWLFjg+h599FEge8v+Xnzxxa49YsSIlF7D0mQq8m+0VLd8S82aP38+EKRkWToWhItEyLrttddeCT3OzrkV44AgvevCCy8EwumTN9xwAxB810D+p2ZZeVnbRf2XX35xxy655JIyj1+yZAkQuyCHfXb6n42WZtiiRQsg/P8iF1Ozvv76a9e29CH7vPNLbNs5+PTTT12fbdHgPy7fWYle07Zt2zKPsZ3p/fRfS2W1ggAQfK9Wq1Yt7eNMliIiIiIiIiISuYxFRD755BPXtlKWNgPoL5hLZDbQ3yjMZiRiLbpLZMPEfGQLaAt9EzSbpfIX6seLCHXu3BkIoib5uBlhZbHSnfanbT4FQclBf4PI6667DggWuccyffp017boy2mnnZamEWcfm73yP8us/KJfDtqfhS4UFvWItSA9HisoIQF/Vnq//fYLHbMoCASLiffYYw/XZzO0u+22GxAU/YAgIpINi2FTZd8ZH3zwgev7+OOPAXjyySeBcOltKxTxxx9/JPX32O8cf3NNm9H2MzgsclKzZs2kXj/bfPPNNwA0b968TJ99V1h5Xgginn6RCSsCYt8dhcA+960su33+AaxevRoIIkV+IQCLKPmFI/zjmaaIiIiIiIiIRE43IiIiIiIiErmMpWbZghqAzz77LKXXsLDpI4884vr8RbEQri/v7+RcSHbZZZcyfXbuCmkn3FmzZgFBylUsnTp1cm1L/8mGxVz5yFI9IHif2v+b//3vf3GfW3rRXq6qXbs2APXr13d9gwcPBsJpC6VZGgMUZmqWSSQly4ojAHz55ZcA3HjjjUm9Rj7zd543tvv64Ycf7vr69u0LhPcdsc/Gv//+Gwjv3GwOPPDA9A02Ar/99ptrW8qZpbsky//utWvPN2/ePCD3zlFFNWnSBAhS6SFIybK9j9a1T4gV4rA0uGxKNaoslpo3c+ZMAO644w537JBDDgGCojq2v3kxJp8AACAASURBVBfAEUccASS2f1UmKCIiIiIiIiKRy4ryvfH45elKs0VKd955Z7mP2XrrrdM+plxTesdNCBYvrV27NurhRMqfcbESk7HY7MHYsWNdX7KREIsu2QzNlVde6Y75ZQdLq1GjBgBDhw51ff6ixUJgC7KtBOW6IiINGjSo9DFVhEU6AAYMGACEI782fitnGqt8ZzpYtHjQoEGuL5tnDl977TUgmN2rCIuE+O9Dm6HO9zKy8diCV/vc8YtAWLn8DTfcEIB+/fq5Y926dQPCn4tWZvv4448H4IUXXnDHrOS+H+3LBf735VZbbQWEIyK33XYbEJSNbd++vTtWukiOZR4ANGzYEAgXB8j2z7F0s7LmtsO3X4zIPqv8wibxWPaLReMKiUUq/Yil/aaz38XffvutO2bRow02yM6f/IqIiIiIiIhI5LLz9sjTo0eP0H9byTYIclVjlert3r07AHXr1q28weWIxo0bA+HSjG+//TYQjiZZbno+sNkAK10MweZRsey+++5AeOar9LVzzz33uPY///xT5jXs2kx2zZOVrM2FKMiaNWtce9KkSUAw01eRWWybWbXyl7H4szmWY5yt/Fljf+PGqNmGc7Gu12zUtGlTIPy+vfrqq4H419dNN93k2n6JTwhvVnrzzTcDhb0uxEq/2szztdde647Z+beccv+8/vnnnwC88847rs/W1Nn6hzFjxrhjpb+7c4X/3rV1gra5IATXYSKzy1ZmFYLz7UdECsFLL73k2hb1t8/7IUOGuGP2OyUW+97xN3ds1qwZoLL6xrbEsOi3v34k1jqwbKKIiIiIiIiIRE43IiIiIiIiErmMpWb5ITYL09mfPtt1/dVXXwXCpQNjPd5KWfphKfmPX5rWQun+Tpv5xBYL+wsJ/d1xS7PratSoUa7P370a4PPPP3dt//qtKNuZ1194lm0sNH7kkUe6Ptt12S93mYxff/3Vte+9914AHn300XIff8ABB7h2vXr1Uvo784WfjlCnTh0gXNK3ND/9xtJtYqW0ZpotJvfLrttO1kcddZTre+aZZ4DYJVEtrevhhx8GCjsNKx5Lo/I/d4477jggdvEEW3j93XffuT67Di1N88QTT6ycwWZI6Z3mJXnTpk1zbVucbsUQ2rZtW+7z/DRpW4Dtp0sPGzYsrePMdfbdvNNOOwFw1llnZXI4Scm+byIREREREcl7GYuI+GXbSs/M+f/dpk2bco/F6rvmmmvSNcS8Zuc/VmnffGAzLn7Z0h9//BEIZt9jKS4ujtlOl5EjR7q2lZY+9thj0/73pNull14KBFEQ36pVq4BwqeyqVauGHuOXibaZ2CuuuML1+RtrQjjiZBttjh8/PqWx5yM/WmeLQQ877DAgXLbR+KWhbXbR3iPZxGbW/UWsFh3x3zu2AN0WYXbp0sUdy4WiD9nAFgffd999rm/27NlA7Peavb/9961lIFgpWylfrVq1gPRG03PBW2+9VabPopT+gv/33nsPgPfffx+IXezA3wRXi9Thq6++cm2LIFkEKhs/38ujiIiIiIiIiERONyIiIiIiIhK5jKVm2Y6lEITY/FrdibBdYP268ZbGIfFZmtKCBQtcn+1qnU/80G/Pnj2B8G7r/kK6VNjiWgjSGrbffvtyH+/XqPfTE7OdLSqMldZm9fGbN2/u+vydxQFWrlzp2lZ4Ih7/fTxv3jwg2OtFwoqKigB45ZVXgPB+G7aDsc/208nG82ljf+qppzI8ksLRtWvXMu1x48Zlajh5y87t448/nuGRRKtRo0au/frrrwPBruiJFgO48sorgXBaYCGz3eRPOeUU12ffA35Rj1yhiIiIiIiIiEQuYxGRFi1auLbtXuqXWk2E7Q4uifEXJW600UZAYS3s3H///YFgQSxA3759Afj0008B6NWrV5nn2SxM69atyxyzaAAEMxL56OCDDwagT58+rm/EiBGhxyQS6SiPRa6sjPIJJ5zgjuXzeU0nW7g9ZcoU13fVVVcBOp8ikhkXX3yxa1tpWSt+4mvZsiUQZMb4JbxLl9IvdLNmzQJg7ty5rs/Kt+fSInWjiIiIiIiIiEQuYxERn+WfW3nGc8891x2zO+gZM2ZEP7A8Y5tVAbz55ptAbt49V5T/b7Z1DfZnrJKBEsxI+ZtI2eZlTz75JAANGjRwx6ZOnRp6/l577VXmNf2yxTZLb5vzSer8Epe2bkREMqtdu3ZAEH2H8JrBfOVnDVx00UWhPyU1ti2BZSoAbLPNNhkaTcUpIiIiIiIiIpHTjYiIiIiIiEQuK1KzjC0UjrVgWCqu9OJikWT55ZCbNWsW+tN38sknRzYmEZFst/HGGwNQt27dDI9Ecp2VZb/nnntc33rr5W5cIXdHLiIiIiIiOSurIiIiIiIiIhLbt99+m+khpJUiIiIiIiIiEjndiIiIiIiISOR0IyIiIiIiIpHTjYiIiIiIiESuSklJSeIPrlJlJbC08oaTM3YuKSmpncwTdO6cpM8d6Px5dO2lTueuYnT+UqdzVzE6f6nTuUudfq9UTELnL6kbERERERERkXRQapaIiIiIiERONyIiIiIiIhI53YiIiIiIiEjkdCMiIiIiIiKR042IiIiIiIhETjciIiIiIiISOd2IiIiIiIhI5HQjIiIiIiIikdONiIiIiIiIRE43IiIiIiIiEjndiIiIiIiISOR0IyIiIiIiIpHTjYiIiIiIiERONyIiIiIiIhI53YiIiIiIiEjkdCMiIiIiIiKR042IiIiIiIhETjciIiIiIiISOd2IiIiIiIhI5HQjIiIiIiIikdONiIiIiIiIRE43IiIiIiIiErkNknlwrVq1SoqKiippKLljyZIlFBcXV0nmOTp3/0nl3IHOn9G1lzqdu4pZtGhRcUlJSe1knqPz9x9dexWjay91uvZSp98rFZPo+UvqRqSoqIiFCxemPqo80bhx46Sfo3P3n1TOHej8GV17qdO5q5gqVaosTfY5On//0bVXMbr2UqdrL3X6vVIxiZ4/pWaJiIiIiEjkdCMiIiIiIiKR042IiIiIiIhETjciIiIiIiISuaQWq4uIiEhhePfdd1170KBBALRp08b1HXTQQQA0aNAg0nGJSP5QRERERERERCKnGxEREREREYlcTqdmvfPOOwA0atTI9e20004AzJgxA4B69eq5Y9WrV49wdCIiIrnrsccec237Tp0+fbrr69ixIwBTpkwBYMMNN4xwdCKSDxQRERERERGRyOV0RMSst15wP/XNN98AwY6Op59+ujs2ZsyYaAeWASUlJa593nnnATB58mQAvvrqK3ds8803j3ZgEXvjjTcAuOWWW1zftGnTgPA5qlKlSrmv0aNHDwA6d+4MwCGHHOKO/fzzzwBss802rq9QZwPXrl0LwNtvv+36BgwYAMDcuXOB+OcZYPbs2QBsscUWZY7tuOOOAGy77bYVH6yIrJO9l4cNGxb3cRYl+fPPP4HC/QyUirPv5auuugqA1157zR3bYYcdANh7771d34knngjA9ttvD8BGG20UyTgl/RQRERERERGRyOVMROSPP/4A4OKLL3Z9TzzxxDqfd/DBB1famLLR33//7dpPPvkkEMze+zMMRx99dLQDi4hFQo488kgAfvnlF3fMZuXXNTtvxo0bF/pzn332ccfsnNapU8f1lZ6RGTlypGvvuuuuif0DssjKlStd248slfbbb78B4X+v8aOV8Rx11FHlHjvwwAMBmDlzpuurVatWQq9bCH766ScAGjZsCMD8+fPdMUWREnfPPfeU6evTp09Sr/HRRx8BsMcee6RlTFH74YcfAGjVqhUQ/vyMpV+/fgBssskmlTswyXv2vdy/f38AVqxY4Y5NmjQJCCJvAHfddRcA//77LwAffPCBO1azZs3KHayklSIiIiIiIiISOd2IiIiIiIhI5HImNevrr78G4Pnnny/TFy/9o3fv3q697777AsFC9nxUtWpV17Z/py1S//bbbzMypigVFxcD604pSIW/y7BZunRpuY+39AaAl156CYCdd9457eOqLLZrMsT/d1Y2S7fzS3HbYviBAwdmZEzpYter/76NtWA/HitKUa1aNQA23njjNI0uf/lpWMmmX8VTv359IFwQI5fcfffdAPz444/lPuaEE05wbVtYnGgKZqGw//9r1qxxfU8//TQAEyZMcH22BYF9tyT73s9Vlmpv/34I0gIXLVoEwP333++OWdrz4MGDXZ8VirHfOf737VtvvVUZw84p33//vWvbOW3Xrl2Zx9m12rVrV9d36aWXAsFv5sqmTw8REREREYlczkREbEGmLRJORfv27YHwInd/M8R8Ywv7bVOq9957L5PDiUSsO/5M8csl77777gB06dLF9Y0ePRrI3rKDfjTx8ssvL/dxNWrUAILZVICbb74ZgMWLF6dtPP57f9asWQCcddZZQO4tyl6wYAEArVu3BsIL/U866aR1Pv+LL75w7alTpwIwdOhQoHBmVVPx8ccfA+mJgnTq1AkIz8TmIr8QRayCE6U9/PDDlTmcnON/Lr3wwgsAjB07FggKxpTHFvn7EdF8888//7i2LTC/7rrrAFi1alVCr2EFUZo0aeL67FzXrl0bCJePt+hKIS1anzdvHgATJ04EwpG333//HYhdqMf6/Pf1nDlzgHBU6ogjjkjziAOKiIiIiIiISOR0IyIiIiIiIpHLytSsIUOGuPagQYPKfZzVj07Ud999B8CSJUtcXz6nZu25556h//bD7naON91000jHVNns3zhixIgyxyyFwtJ5fLaYK9Z+GZbS5oeYk2X7uzz44IOuz/6u7bbbLuXXrUxnnHGGa/fq1avcx9lCVT8l6JhjjgGCnZnPP/98d8xSWj788EPX5+9Zkogvv/wSCFI2cy01a8yYMUBwjhNJx/JNnz7dte3a6tixY5pGl79sMbkvXoqV9eXqviCJsM8+gOXLl4eO+YvQ/YXChcx+R9xwww1AkIYFwT4XVljD//1i71NLS4Jgd/Dq1atX3oAz7IILLnBt+17ef//9geBzEIK9v+wcPv744+6YFSfx9xGxggpWEODqq692xwolJeuVV15xbfsOsfew7ccC0KFDBwDWrl3r+k499VQAli1bVuZ1rbCRnwJcmRQRERERERGRyGVlRMRfUJNIWUB/Ua3dVc+ePRsIFgT7pkyZ4tq2w3g+z0hYeTZ/NsHKybZt2zYTQ6o0PXv2DP2ZKJvBijUzbTMzfilG48/02wK5RFkRgXSWD00nP2Jx6KGHJvVci47EimhOmzYNgG7durm+Z599dp2v6b9HreBErs5U22J7//pJxjfffOPauVoqNp1sEToEJd7PPffcMo/L9Z3P08lmQl9//fVyH3Phhf/f3p3HWzXvfxx/dTOUIaGSqSJcQ8ZCIkmRa8qQjDdThi6Jm9LP1EBSpuQSkWQIGZIQiroISV1kvMqQDKluSpTx/P7o8fmu7zpnnX3O2cPae+39fv5zlu/ae59ltc7ee30/n+/n09ttpypYUazsevGLoNhssS0A9s/LGWecAUCzZs2A8CJ0e54fEbHIQDGyyJGfnWDf1fzCJuXZAn47lxB8Lvtlpa1suZUr9yMApcI/txYJue+++4BwOd611lrzVd/P6rBIlWXH+C0PorIcckkRERERERERiV1BRETsTszKr1mJN9+mm24KhPPpbZbWSoVCMAORqnnfU0895batsU4xR0SiSrb50RFJLVVJ4OOPP95tW/6lNQPy14NElZ22Wf1CjYjUNAry66+/um0rt+ifH2MlG6MaREapV68eEM4Z9ss4JoU/45SqnGJ1+JFem0FcZ511Mji6ZLLry6KLvqiIiCIhAbuG/Ohaea1bt47rcArSsmXLAGjbtq0bs3WVp512GhBukFzTv2f72y1G9l3MPyczZswAgu8f1oS1KrVr1wZg5MiRbmzixIkAPPPMM0DxrXetDrs+fXbe7ZxB0ODR/7ew9SVRzZ8tUm9rmHJNEREREREREYmdbkRERERERCR2eUvNsoVMEJSdfPvtt4HoBeoWKqpqwZylfZTiwiWJnx9attSYI444Agg6nFbGLzeYZJaS5Zdp9EPo6bBu7RB0J953330zes1823DDDd12gwYNgKAAgpX2hGBhYSqW+gZwwAEHAEHJypq+VhLZ4vSolKyo0t1SkV/6U6Ltt99+oZ+ZsJRdX03LdieJdTy3MscAffv2BaBp06ZAuHyvlXz3U4qMfVaOHz/ejdn1m41/m6TacsstK4xZqne/fv3cWP369YHU6dBnnXWW2x4yZEi2DrFaFBEREREREZHY5W267I033nDbc+bMydrr2qzjFVdcAcDgwYNTPn7gwIEA3HrrrVk7BilNtgDvqKOOqvQxnTp1ctt77LFHzo8pDvb/nWkUxPfoo4+67aRHQqJ0794dCGatvv/+e7fP3rP8qFAq1nDzhBNOAMKN53bZZZfMD7YAlW9MaE0JIXqRugSseei8efOA1OWf/QIRVp7cf7yVWI2axZYwv5FyKfFLQFsTPXv/8z8rLTpkRRSsDC3AmDFjgHBJ+VJpWpjKiBEj3LYVaLLy5X6p46+++qrK1/Ib4vplp+OgiIiIiIiIiMQu9oiI5ZO/++67buzPP/8M/fTL4dndXU1Ls5V/zcpev6qISTGwGax0S4VK5X766Se3fd5551X6OFv31LNnTzdWLKUbbR3CwQcf7MZefvnljF7zzDPPdNuvv/46AFtttVVGr1lI7DqwtQ533nmn22eRpR49egBQp04dt+/TTz8FwrPSNvNlEZFijYKk4q8VsZK+HTp0CP0Ele8F+PzzzwFYsGABkPpzYezYsRW2/WvPZrvtHD/00ENun3/dSqB9+/ZuuxRKbvtrfhs3bgwEJXcnT57s9lmDW38tnbFIiKIgYf65snNpawT9v1P7PI7K1rjkkkuAcLZG3BQRERERERGR2OlGREREREREYhd7atYtt9wChMuDlS/X65forWlKlnWJtNf3X9tSsiZMmJD26yeRUrKyw0/zmzJlCgDDhw93Yy+++GLo8X5p3xtuuAEIShQWk7p16wLhBeb+osLyOnbsCIQ7sZdnC+8AHn74YQD69OmT0XEWEktbGTVqFADdunVz+6zssy1C93333XdA+G/ayp4XS/GD6rASvVaq0mdpWqlK+2pBe3bY5+1TTz0FhN8DLFVwvfXWi//ACsSKFSvc9uzZs4FggT9EtyooJf7nYZcuXYBwSV/z4YcfAhWLVEjACkbYT78AyuWXX17h8bY4/Zprrgk9Lx9K+69ARERERETyIvaIyKxZs6p8jN/s8I8//gCyc7f2+OOPA7DFFltk/FpJt+uuu+b7EBLHoiAAhx9+eJWP92d7omZui42/kHD//fev9HE2izpgwAA3lqqBkpXi9ku0Nm/ePN3DLCg2I3rggQe6MX+7vOnTpwPhwgC2ALuUZlctomE/77jjjgqPifqbs7FSXsDeqFEjIPh7tbK8Ufy/YztPo0ePrvTxflO05557DghKr0LpRUf8oh1W5txv/FqqbCG1n/1i15VlFvgFPKwAhV9OOlWZ/FK2fPlyINySwhoZ+sURLDvJMhryqXQ+uUREREREpGDoRkRERERERGIXe2qWpVdMnDix0sdY+gEEC7uiQrpWP996jUCwmLP88yHoXbL11lvX6JiLkdXzlsrZouGBAwcCsGjRopSPr1+/PgCfffYZEPTXkDA7L6effrobszD8smXLKjze0jNTdYAuFamKAJSyqMXnln7lL9S0Bez+otdSu66aNWsGBMUNUvX8OeSQQ9y29b658sor3Zj1fnjzzTcrPNdSof2/8+qktBYT/7uJpU5aalwpe/XVVwEYOnSoG7NUIrvm/BRVSxHs1auXG9thhx2A0kutrMqcOXMAuP76692YFTexHlUADRs2jPfAUlBEREREREREYhf7lK1/l1aeLcC8++673ViqxW1ffvklACeddFKlj7nooovcthZoS2UsinHjjTe6sbfeegsIOhFH8buRDho0CICNNtooF4dYdPwZmVQdhm2x+jbbbJPzYypE1ikXYPz48QDss88+bqwUujOnw2ZK/YXpFhHxCx+Umnnz5gFBOdlU/IISdu1ZV2wIlzSvzIgRI9x2qUVE/MI7bdq0AUr78+Hnn38GgtKxPj97BcLl7y261qJFCzfWu3dvILguS60QQnl2rdl3Er/Eu0Uxy5/jQqGIiIiIiIiIxC72iEi7du0A+OCDDyrsmzp1KgA333yzG7OZqyeeeAKAwYMHu302GxNVttJmpxUFiebPskqwtuiuu+6q9DEbbrih27YZhgsvvNCNWZM6SW3p0qVA0PAMUq+/sfOez4ZL+bR69Wq3/corrwDhyN3aa68d+zElwSeffAKE8/QFNttsMyCYoZ88eXK1nmdN5WyNCaQuGb3++usDwcy1iDVftZLRO+20k9tn10sUu+YsMgJBefxnn30WCH+elIqVK1e67b59+wLBmkr7rg3hyGYhUkRERERERERipxsRERERERGJXeypWVtttRWQOqR73XXXRW5X9jx/bPPNNwega9euGR1nsZs2bRoQdCwtdVHlP8vr16+f2+7cuTNQXCV6P/74YyBcbrO8cePGue2NN944tM9PTbPw8G+//QYEXVwhWFxoi2YltRkzZlQY69KlSx6OJD+sa7qfYmWfC1GlOy0lyy/RW56/gL3UWKrjI488AoQ/K1944YWs/R4rIuOXAC4V1kX9+eefd2PqBA7vv/9+6L+tEAlUL/W2bdu2btuKdFiqbynyiz/Y54S9J44dOzYvx5QORURERERERCR2sU/nzpo1K2uvZTM7flnPCRMmAOEFdaXKFn+1bNkSqF65xlJjM/dfffVVlY/1Z29s++yzz3ZjqRbbnX/++QA0b968wr5Ciar8+OOPQLCgMIo1kYriz84sXLgQgPfeey+tY/EXuJby7DXA66+/XmGs2JuyWhQE4IILLqiw38rw1pQVP6lOBLTYbbDBBgA89thjbsxmUe39bcWKFTV6TT/64UdBS83MmTMBWLVqlRuzxcQS2HvvvWv0eL/hbSkW3LECTdYQ0v9ssPLFQ4YMAZL1GaGIiIiIiIiIxE43IiIiIiIiErvYc0Luv/9+ILwg9qmnnqr28/3OkLZg+Mgjj8zOwRUZW/zl978wkyZNArRYfc6cOUA4hF4To0ePrtbj/O7CxtIYhg8fDqReXJsEzz33XNrP3WSTTYDgmu3Tp4/b53dgLyXWKfeaa65xY0cccUS+DidWUelY6br99tvdtlKyKvJTSu38WMrpww8/7PZZR3W/4/Xy5cuBoHjCiSee6PbVrVs3R0dc+KIWClv/llJmPY9soXnjxo2r9bwlS5YAQdoRBGlKTZs2zeYhFrSvv/4agDPPPBMIf7e75557gOB7cZIoIiIiIiIiIrGLPSJiC2rs7g2CWZWoDrhW/q5Ro0ZAeLGvvZakts8++wAwffp0N+Z35CxltljOZvuGDRvm9lkZ0AULFmT8e6yknn/ebaH8QQcdBATd3fPFSl9feumlANx2221un5WjzIZLLrkECJdrHDhwIKDu9D4rHlCrVi031rp163wdTqz8KEZUdMT222eGX9BAUY/MWdTDz0Dwt6V66tev77br1auXxyMpDBYxu/rqq4Hw3619Lphvv/3Wbdtnkh+9sywD67BerBYvXuy27fuKjQ0dOtTt69SpU7wHlkWKiIiIiIiISOzyVjd0o402ctsvvvhivg6jJFgjvnfeeceNpWpaV4oOPvjg0E8IIiEfffRRhcdbycsxY8ZU+pqjRo1y20cffTQQzHJDkDP82muvpXvYWWXNRm2WZbfddnP7unXrVqPXshnrqPKMe+65J5C6qanA008/DQTRYAhmBoudH9VIFeFQ9EMKka099Ne3Ra3VLDUWFZoyZQoA7du3d/tOPfXUSp9na5auv/56N9agQYNcHGLBWL16NQDHHHOMG7NIyLHHHgvAueee6/ZZOe4k0jcBERERERGJnW5EREREREQkdoXR0llyyhbM2cJ/qZ4mTZqEfvpsYZhfdKE6okrRFuoiMz9UnipsLrnVsmVLt+2XThWRwjJu3DggSIO2DvUS1qJFCyC8EFuCwk0ABxxwABCU7AXo378/EFxXfsGXJFNEREREREREYqeIiIhIAbKmjn5zRxEpXIsWLQr99ymnnJKnI5EkseaMgwYNcmPt2rUD4Pzzz3djFkkqNoqIiIiIiIhI7HQjIiIiIiIisVNqloiIiEiGrDt4+S7hIqlYT62bbropz0eSH4qIiIiIiIhI7GqVlZVV/8G1ai0Gvszd4SRG07Kysop1WFPQuXNqfO5A58+jay99OneZ0flLn85dZnT+0qdzlz59X8lMtc5fjW5EREREREREskGpWSIiIiIiEjvdiIiIiIiISOx0IyIiIiIiIrHTjYiIiIiIiMRONyIiIiIiIhI73YiIiIiIiEjsdCMiIiIiIiKx042IiIiIiIjETjciIiIiIiISO92IiIiIiIhI7HQjIiIiIiIisdONiIiIiIiIxE43IiIiIiIiEjvdiIiIiIiISOx0IyIiIiIiIrHTjYiIiIiIiMRONyIiIiIiIhI73YiIiIiIiEjsdCMiIiIiIiKx042IiIiIiIjETjciIiIiIiISO92IiIiIiIhI7NaqyYMbNGhQ1qxZsxwdSnJ88cUXLFmypFZNnqNzt0Y65w50/oyuvfTp3GVm9uzZS8rKyhrW5Dk6f2vo2suMrr306dpLn76vZKa6569GNyLNmjXj7bffTv+oikSrVq1q/ByduzXSOXeg82d07aVP5y4ztWrV+rKmz9H5W0PXXmZ07aVP11769H0lM9U9f0rNEhERERGR2OlGREREREREYqcbERERERERiZ1uREREREREJHa6ERERERERkdjpRkRERERERGKnGxEREREREYldjfqIiJSSZcuWATBv3rxKH7Ppppu6wWzMeQAAFflJREFU7W233TbnxyQikmtlZWUAjBw50o1dcMEFAIwZM8aNnXHGGbEel4gUH0VEREREREQkdomJiHzzzTcAoW6VxxxzTJXPs5kdgE022QSAjz76yI01atQoW4dYNO69914Aunfv7sZuuOEGAHr37p2XY8qVYcOGAbBo0aIK++bOnQvAlClTKn1+kyZN3PYLL7wAwI477pjNQxQRicUPP/wAwMCBAwEYMWKE2/eXv6yZt5w2bZobU0RERDKliIiIiIiIiMSuICMiS5cuddvPPvssAIMHDwbg008/dftq1apV5Wv5j7HZnqOOOsqNjR8/HoCmTZtmcMTF4ZdffgFgwIABQPjcXXnllQC0aNHCjXXq1Cm+g0vT4sWL3bb9+/t5z7feeisAf/75Z5Wv1bhx4wpjCxYscNstW7YEgqjdTjvtlMYRi4jkx6WXXgqE14GYddZZB4Djjz8+1mOS0rJixQoA7r///gr7brnlFrf92WefhfY9/vjjbrtz584APProo26sR48eALRu3RqAyZMnu321a9fO9LALxvLlywE4/fTT3djTTz8NhL/TWbbQXnvtBcArr7zi9q233no5P06fIiIiIiIiIhI73YiIiIiIiEjsCio1y1KDTj75ZDf20ksvZf33+AveX3vtNaB0U7P8lCQLY3799dcVHrfFFlsAsMcee8RzYBn69ttvgXAa3uzZsyt9fNeuXYHUBRCiUq1atWrltn/++WcALrzwQiA3124cfv31VwD++OMPN/bGG2+EfmaDlQMFqF+/ftZetxT4xRWuu+46AG6//XYAPvzwQ7dvhx12iPfASoj9fVjKp88v613olixZ4rZff/31Sh9n6VpHH310zo+pUFj6ypNPPlmjx1sKzHfffef2XXTRRUA4Xcge17FjRwA22mijDI84WX7//Xe3balBF198MQAffPBByuda8QRjn+FVefXVV4Hw51sxpGZ9//33AFxxxRUAPPPMM26fXWdRyxneeecdAM455xw39tBDD+XsOKMoIiIiIiIiIrHLW0Rk9erVbtvu4B588EEgPENTU3Xq1AGgQYMGACxcuDDt1yoF8+fPd9tnnnlmpY974oknANhss81yfkzZYAuw/Bmp9ddfH4BddtnFjT388MMAbL755gDUrVu3Wq9vC8L8WRl/hiUp/AVqFsGZOnUqAG+++WZOf7c/M2iRFvs3Evjpp5+AIEIFsPHGGwOw3XbbVXicXcvVjYLYOd9vv/0yP9gi4s/SWmR15syZbszOm41FRQmT8F5g182BBx7oxj755JNKH2/FOIqVRbT9f+tevXoB4ShjKuUjIj4bO/HEEyuMbb311kAQNQE45ZRTgOR85tbEjz/+CED79u3d2H/+85/QY/zokLUN8N/b3n33XQCGDBlSrd955JFHAnDjjTcCQfGFJPv3v//tti2TyM6t/3dtRSgOOuggN2aNmvfcc08gXIgoboqIiIiIiIhI7HQjIiIiIiIisctbapYfUho+fHjWXtcWFFvfkcMPPzxrr11MbIHl2WefXeljTjjhBLftpzMlwf/93/9VGLOw7mGHHZa11//tt98yfq18ateundu2NDP76af/mCOOOMJtW8pAdQoY+AUirr76aiC8GHHSpEkAnHTSSdU+9mJlBSSsDryfsmCplCtXrnRjds5SFVqIsmrVqoyOs1h8+eWXQNCH4JFHHnH75syZAwQpN5C6f1WSemzYv3+qdCz/89MKlhQrKzIS1b8i17766isA+vTp48buvPNOIPwZbQu5k5pWVD4ly39v22233QAYOHAgEP5ssjQtP23SFlmn4vfDGDRoEADbb799WsdeiPzCTltuuSUQFC3ZZ5993D5LO5w+fbobGzVqFADdu3cHgjTEfFBEREREREREYhd7RMRm8u64444aPc+iJn4JVSupd9ddd7kxW4hkpYAlmnVF92eqjZVSveqqq9zY2muvHc+BZYm/6C9b5s6d67bHjx9f6eNsJisJdt11V7dts0c333wzAG3atMn49a1L7gsvvJDycdtss03GvyvJ/MXNNjNr729+BM9mr/1Z+WuvvRaAddddN+fHmST//e9/gaCwBASzolGlLY0/Y2oLhqMiIrbIs0uXLm5fkyZNsnLscfDPQXkWCfGjQ8VeSCLqszDKiBEjAGjcuHHGv7Nnz55AuBy3scXEl19+uRv729/+BoTft5PEvp9ZJMRm8QFefvllICjIEcWP3l1//fVV/r5p06a57d13371mB1vA7rnnHiBcjMciZ34kxFhLBiu+AUG3dSsWFXc3dZ8iIiIiIiIiErvYIyI20zJjxowK+yw3vWHDhm7MZuUtj82fmd9///2BYJYLYJNNNgGCkpf2PAjuIgVmzZoFROc72zlP2rqQXLGcfT93eOnSpRUe17lzZyAoxZgE7733Xk5e19YgHXfccUB4TZjxy0UnpVFmttkao5EjR7oxi/BajvRjjz3m9kXNYvvvlzVRTE0k/XLwlmv+xRdfANGldP33PYt63H333QCstVbwsZi0SHBNRP3t77jjjkBQCrqqKIid92uuuQYIz3CfeuqpQHKa9E2ZMgWAG264wY1FrV+1zwGL8mby/2fvj/Z5UoylelPxr69UM/IWWa/u+h1bX2fvBcVm3LhxQOr1aj6L8g4YMMCN2XML4RwpIiIiIiIiIrHTjYiIiIiIiMQultQsC5EDvP/++5U+zlIMvvnmm2q9rnXBjuqGbYvV/YWKpc7vQBrVAdbK9dpi2VJnKVnDhg0DgoV2Pr9TqYWN69Spk/uDKyCWXuSnehx99NFAeDFdebbIGkpvobWlC11yySVAuHiHhcpHjx4NhBdKn3feeUB4cX+6pTz32muvtJ5XCOxv0xYO+yk0VgrVrqmdd97Z7TvqqKOAcJno5s2bA6Xzd2sd1SdMmFBhny2M3mCDDSp9vpU7hiC9Mir10ope+J/5hXyOLS1q6NChbswWhfupk7ao3QqWdO3a1e3LRRqavyg+6emUlq5nBV2soATAP/7xDyB4L/Q/E84991wgnKZani3kh+DfK6lljqtiRST8a2Py5MlA8Hlx4oknun32PukXn7DPEL8kf74oIiIiIiIiIrGLJSLiLyZPFaHwy8Vm6s033wRS30GXiv79+wNw3333uTGLhOy3335u7N577wXCizVLmZW3i2qOaGyGB6BevXo5P6ZCZLPR/fr1q9HzevTo4bY33HDD0L7WrVu7bVtMnPTZQJ9Fj6LKmFuzLyvp+cQTT7h9tmjz0EMPdWOpZpmtbOOCBQvcmM3yppr1LnQWZe/duzcQXV7XFp/7Tb+sIEops4Xodk34C4ajSn8aW5huDUkhOhJiPv/8cyDchC4Jateu7bZt0bPfLPS0004Dgs8HPyKSrlSFdPzrN0mFUKJYU+GpU6cC4SIl9v3E/n7tewsExU+iWIGiUoqwN2rUCAgXY7Lvb1Ya2S9vHJUBY6XG81m21+hdWUREREREYlcQU982o+rf3UnmLJfXZhoWLlxY4TF9+/Z124VwZ5xvq1atctu33HJLaJ+fg2pNgPyIUqnym0zVhM0oRnnooYfctuWY/+tf/3JjSZ/ZtuZa//znP4Egnx7g1ltvDf2M8tZbb7nt8pHkZ5991m3bufNnpefMmQMku2SyRR9tVs9ft2CuvPJKAJo1a+bGrOR7KbNGecZfQ+M3cyzPIsMPPvhghX323tiyZUs35s9QJ52/9mPSpElZe12LevrZCsaiMFYauZhYmWhrYgjBuq0xY8aEflbGIiFWdjnJ72fpGjVqlNu2hoYW8fTXbFpE6d1333Vjl156aRyHWC3J/jQXEREREZFE0o2IiIiIiIjELqepWWPHjgWiw46bb765227bti2Q3S62nTp1AoLUB4CbbrqpwuOsDGQxssWaVs7SZ6U7bWFsqfvf//4HhBcBz549GwjSgLp06eL2KcUjMHjwYAAuu+yyaj3ezrV1EIegdKMtQu7WrZvbZ4/zr+OJEycCyU3RsuO2BYV+Odk33ngDCBYTR71v+f/f1mU3ipVRbtGihRvzu18nVYMGDYBgYaZfctwKlFi6lp+65qeClJKff/7ZbT/wwAOhfVa8oCp+GX6z1VZbAUEqpaX9QXGlZuVKx44dgXAZW9OnTx+gsEsep8u+67Vr186NWUrp3nvvXenz7O8egs72pZiSFWXfffcN/fRZAYbqdmKPWzI/xUVEREREJNFyGhGxu6+ouzB/xjOXd7T+zGHUcSR1RrUyjz76qNu2xkxR/9/Tpk0DKpZNLVVWItWiIACdO3cGgiZy/mJ1CVgjMPsZZf78+W7bogB+pNQWg7Zq1QoIz+pYCc3nnnvOjS1evLjK35kEVirb/r/9bb+JmrHSqQMGDMj4d8+dOxeo/ox4IbKSzv4CaiskYVGgZcuWxX9gBcaP/JdvMmpNCaP4UUj7zLAoiD9mf7+pCizIGnbOIHhftM9o/33AL2FerH799Ve3HVUEoTxrbAiKhFRl5syZbtvK9/qfl4cffnjsx1SZ4voWLiIiIiIiiaAbERERERERiV1B9BGRzFnH5RtvvNGNWTjeFir5HcKVkrWGLa60Ds1+wYTjjjsOUEpWJqy78sUXX+zGLH3Dr81fnt9B2FLj/C7O1gE6qn9E0v30009AUGijQ4cObl82UrJMMSxaj+J3WQc45JBD8nQkyeD3S2rTpg0QLG6390WAH3/8EQgv/t92222BYLH6iy++6PZZrwhLPyx1dk793l2WmmT9cGwBdrH77bffAHjppZfcWPm0PusTAsHn8urVq2M4umT7/vvvgfD7nqX++X+fhUQRERERERERiV1Opiq++eYbICjrGSfrIGyLtv1uzObUU0912127do3nwHLESqFaGUC/c6a57rrrgMLqpFkoTj75ZCCY7bOZKQgXVJD0DBkyBICVK1e6sS222KJGr2HlHNdZZx03tnDhwiwcXWGyjri//PILAH//+99z8nv8GcdC43eat27dfqnPVMqXjd1uu+2yd2AJ5UcldtllFwA++OADICib6o+tt956QFDEw1e3bl23PWLECAD69esHwDbbbOP2vfLKK0Bxlp9Nh13Tfoljm6m2iHGqKHExufPOO4FwpNycddZZQPDZAUF57mJ+388WK0ZhkXUI3jv9Mu6FRBERERERERGJXU4iIjbjaWUUU5UHzDaLhKSazfZnVrPZRDEfrOlZVCTEHHvssXEdTuI0b94cCBpKWTQPgoaQ55xzTvwHViSszK7NjkIw4zVw4EA3tv3224eeZ039AB555BEgXOqx2Pj/bzZz2rRpUwCOP/74vBxTPvnRW5vB9/Pny0fVLKIJ4TLPEDS3LWV+VMLKdlr0wyJvEOSVp1rX0bNnz0r3WXlpgE033TS9gy0i/jXbv39/IFxKeYcddgDCDU2L2ahRowAYNGhQhX1WvtfWZq677rrxHVgRsLUhhx12GBBu22Al8wuVIiIiIiIiIhI73YiIiIiIiEjsiqKu3uOPP+62/c6b5dlCML8kYdL98MMPle6zEGexlunMhtGjRwNBqocVOwCYMGECECzc3Hnnnd2+PffcEwh3sj/ggAOA6PNtZQejFn9G8QsqJJktNH766afdmJ2zZ555xo01btw49LwvvvjCbf/xxx8VXrdQyxCmy78ubNvO2frrr5+XY8onK9kMcNdddwHhjt677747EHRN98s4W0pC+RQ3WcPKuFvakBVHAFi0aFFar2n/RieccEKGR1ccrFSvX07frsu//CWY/7V/g0aNGsV4dPH69NNP3fbll18OBH+3p59+uttXPiXLziEEZeCtXLRUZJ8Xtljd/0y1cveFShERERERERGJXd4iIrYQGOD5558HYNKkSUDqspK2OBuC0qv+7KnNPFuJwXr16rl906ZNA+Cvf/1rJodeULp3717pPov8qHxi5WxB5WWXXQbA0KFD3b7JkyeHfm622WZun5WU9UtfWgMvWwDvs/K106dPd2MWoYtaGFosEREr6+k3orKyjH55wfnz51f5Wn5Dzvbt22frEPPKZq+ssAcExT1s0WEp8hdXWkRy1qxZbmzGjBlAEBnp0aOH22eFD2xRtoTZ+469F40cOdLts8aEH374YYXnnX/++QAceuihbsy2bRbbn+0vZTNnzgSC7xy+M844w20Xc7aCFeDYd9993djy5cuBIEppZXwhXEQIgs8OgLfffhtIfruFbPOjRsOGDQOCyFv5BpGFTO8aIiIiIiISO92IiIiIiIhI7HKammUpUP4iQ+uMaYuV/O3qLCosKytz236dZGNh53vvvReAY445pqaHXfAsnQPCHasBbr/9drfth0QlmoWDLeTrL4yeO3cuECwo9Bdy+gutzUcffRT66Tv77LMBaNWqlRuzdLAGDRqk/z9Q4CztzO8ZYmlI48aNc2PWbyRK27ZtAWjTpo0bi/rbTyJLTWjWrJkbswWdqXo5FDu/w/RNN90EhP82LcVjgw02ACqmdUjV7Bz7KTCWfrX//vsD8PHHH7t91gW7fM8fqahXr16V7vPPd9L7mKVi38HsbxWCwhuWhh/1dzt16lQgKCQD0KRJEyDennRJ4F9nlt587bXXAskqHKGIiIiIiIiIxC6nU242I9+hQwc3Nnbs2Fz+Su655x6gOCMhxu+i7s82QLBIH4pn1jgOe+21V+gnBKWRq7OQuiotWrQASrdbbO3atd22XaMWJSpF1gl8+PDhQHghflSxAwlfQ6kKmkj66tevDwRlt63zN8B2222Xl2NKEis7bV3rfdZawP/8XrFiBRCc72LiFyMxVqL3/fffD/2EIEJupdltsTvAk08+Cejv3lgxBIs6QfDdJVU0rlApIiIiIiIiIrGLJQnZL4k6ceJEIHUjvuqyO78LL7zQjfm51sWqU6dObtsa8dnsgx99ksyUnx0UyZb77rsPgFNOOQUIv4eJ5Nv999+f70NIJMvIiMpGsJYFo0aNcmPZiLYnyQMPPBD6mYqtdYCgTHepsyjRBRdcAMCff/7p9nXr1g0Iyp0niSIiIiIiIiISO92IiIiIiIhI7GJJzWrYsKHbXrp0aRy/smSUWmhXJKlWrVrltq27/MsvvwwEZS1FpLj17dvXbRdzZ3VLH/L/f8vzvxteddVVQLCgvXHjxm6fCu+sYS0E3nnnHSBclMlSs5JIEREREREREYld6XbMEhGJUc+ePd32bbfdBsCOO+6Yr8MRkRhZaV+Lhha7OnXqAOFGpJKZrbfeGoDff/89z0eSXYqIiIiIiIhI7HQjIiIiIiIisVNqlohIDKzHgIgUJ6UhidScIiIiIiIiIhK7WmVlZdV/cK1ai4Evc3c4idG0rKysYdUPC+jcOTU+d6Dz59G1lz6du8zo/KVP5y4zOn/p07lLn76vZKZa569GNyIiIiIiIiLZoNQsERERERGJnW5EREREREQkdroRERERERGR2OlGREREREREYqcbERERERERiZ1uREREREREJHa6ERERERERkdjpRkRERERERGKnGxEREREREYnd/wNslxpLj7sSeAAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# create a subplot array, with each column a different digit, \n",
"# and each row a different instance\n",
"fig, ax = plt.subplots(nrows=5, ncols=10, \n",
" sharex=True, sharey=True, \n",
" figsize=(14, 8))\n",
"ax = ax.flatten()\n",
"for i in range(5):\n",
" for j in range(10):\n",
" # reshape into original 28x28 pixel grid\n",
" img = X_train[y_train == j][i].reshape(28, 28) \n",
" ax[10*i + j].imshow(img, cmap='Greys', \n",
" interpolation='nearest')\n",
"ax[0].set_xticks([])\n",
"ax[0].set_yticks([])\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Some of the digits in our dataset are challenging to categorize even for a human. For example, in the set of 5's displayed below, the first and seventh images could easily be mistaken as a 3."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAyIAAABXCAYAAAD4SvGxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAFZNJREFUeJzt3Xm0VVUdwPEvlJlJjpCRJeRQoakoWk6lYlKrNK3UNKQ0XRU5LMUhIc0JccSlpmYmikJZRqmkoi5tZTmkJlZLW+aqZDCxNMUBR+T1h+u3zz68y+ON593h+/nnHfa573I43OGc/Rv2gLa2NiRJkiSpSgP7+wAkSZIktR5vRCRJkiRVzhsRSZIkSZXzRkSSJElS5bwRkSRJklQ5b0QkSZIkVc4bEUmSJEmV80ZEkiRJUuW8EZEkSZJUuXd25cGDBw9uGz58eB8dSuOYN28ezz777ICu/I7n7m3dOXfg+Qu+9rrPc9czDz300LNtbW1DuvI7nr+3+drrGV973edrr/u8XumZzp6/Lt2IDB8+nD/96U/dP6omsc0223T5dzx3b+vOuQPPX/C1132eu54ZMGDA/K7+jufvbb72esbXXvf52us+r1d6prPnz9QsSZIkSZXzRkSSJElS5bwRkSRJklQ5b0QkSZIkVc4bEUmSJEmV80ZEkiRJUuW8EZEkSZJUuS6tIyJJkprbjTfeCMCRRx6ZxhYuXAjA1KlT09jRRx9d7YE1gddeey1tP/LIIwDMnj0bgNNPPz3t22ijjQCYO3duGltjjTWqOESpUkZEJEmSJFWu7iMiy5YtA+D1119f4WOuvvrqtL1kyRIA/va3vwFwwQUXpH2TJk0C4OKLL05jq622GlDM8owfP743DluSpLr3xhtvpO0pU6YAcNpppwEwYMCAtG+VVVYp/QRYunQpAO98Z91fSlTqzTffBOCll15KY9OnTwfg5JNPTmOvvPJK6fcGDizmhv/zn/8AxTUNGBFRczIiIkmSJKly/TaN8cILL6Ttt956C4C//OUvANx+++1p3+LFiwG4/PLLu/T8w4cPB+CYY45JY9OmTQNgzTXXTGOf+tSnABg9enSXnl/qihdffBGASy+9NI3dddddANx6660AfO5zn0v75syZU+HR1Y/8/ITDDjusS8/x2GOPAfDRj360V45JjamtrQ0o5+Q///zzAFx77bXtHn/KKacA8PLLL6extdZaC4AZM2aksT322KPXj7U//e9//0vbeY3C8mK2Pp+1j3PcyuL6BYqaj4go3XDDDZ16jlVXXRWAnXbaKY1NnDgRgKFDh/bKcUr1yoiIJEmSpMp5IyJJkiSpcpWnZj355JMAjBw5Mo1FuLw3RNg40rCiGB3gkEMOAeB973tfGhs0aBAAQ4YM6bVjaEaRSpcXNl533XUATJ48ud3jx44dC8B5551XwdHVh3vvvReA3//+90CRegVF+lVHIp2wVeRpWF1Nv+rIxz72McC0kZWJlJJIf82tu+66VR9Oj+TpV/fddx9QtKD94Q9/2KnnWHvttYHy+3Dw4MEA7LDDDr1xmA3ta1/7GlCkM0O5cL1VzZ8/P21vvfXWK338xhtvnLbjdXXSSScBsOGGG/by0bW2p556Km1HevQTTzwBlNOfN910UwC+853vVHh09S9PU43vibiWya+jv/jFL/bo7zEiIkmSJKlylUdEYqZtvfXWS2NdiYiMGTOm3XP9+te/TmNR9LXLLrv05DBbWrQ+/vnPf57GLrnkEqD8f5W3dlzenXfe2UdH139iRgWK2fy8iULMtHQkL0iPWbBWm239+9//DvROFOTLX/4yALvttluPn6tZREtVgEWLFgFw//33p7GIGMRY/DmXF+A2gp/85Cdp+6ijjlrp49dZZ520vdVWWwFw2WWXAa03K51nDYwbNw6Aa665pt3jDjroIAA233zzSo6r3t1yyy0AHHrooSt8zLBhw9L2iSeeCMC+++6bxt773vf20dE1v1jSIb538+uVyNZYsGBBGlu+VXLu3e9+N9BaEZFXX30VgAcffBAov+cjq+O5555LY/k2wJVXXtlrx2JERJIkSVLlvBGRJEmSVLnKU7MiDByrjALMmjULgO233x6Ar3zlK+1+L/prRwEiwLve9S4Ann766TR24YUX9u4BN7nvfe97aXvu3LlAx2lV+RosRxxxBFAUL+66665pXzOutJufq0jjyEVYN9Ibcq2WftWRKCbPdZRiFWOuC1JbFMvG52ieohDv6bxwv6OUylqfvfXshBNOAGoXpEea7syZM9NYFKXmn2Otvk5DrJUCxfdyrdQslUXjnVgBPRdr0uTrmL3nPe+p5Liaxb/+9a+0PWXKFKD8Ofa73/0O6FxKNMBXv/pVADbbbDOgnCbdTJ8BUVQ+YcKENLb66qsD5ZTUiy66CCiv6be8L3zhC2l75513BorUwg022KCXjtiIiCRJkqR+0G/T1ttuu23a3mKLLYAiwnH88cenfeeccw5QrPgaj8m9//3vT9tnnnlm7x9sk4jiJChWfj333HPTWLQwjkL//FxGAWd+/vNZxWZ21llnAXDbbbelsYh+HHfccWms1YpclxdF6FBE1b773e+2e5wrn3fO448/DpRnrOJ9e9NNN6Wx5SMcm2yySdqOlqu1IiJRpL3PPvukfb05y1WFu+++Gyi37w3RejeibeqZiLTl7Xtb2cEHHwzA7Nmz01i0hI33af6dcf755wPla5+OopOtYNmyZWk7iv8jgnn99denfW+++SYAa6yxRhqL9/eIESOAcpOKiOzm1ysRFWjWc/7ss88CRQvpiNjl8n97NFI44IADgPL1S1wDjho1qubv9jYjIpIkSZIqVxeJ/JHLG2JhqVzks+WzMc16Z9tXpk6dmrYj0nTqqaemsaiBqBV1akWxQOHEiROBck7p2WefDZRnaFpVzD7lbbRDrYiIkZD28hn9iBDPmzcPqN1KN//si6hHtLDN67OafcG5qL2q1YI42qWqd8yYMQOASZMmpbH111+/vw6n38V76xe/+EUai2hk1I3kr8uogb355pvT2OjRo4Hm/M6NaEfUy0DRrjgyCvKISCyKF5kWsQA1wPjx44Hysg/5gnoqIkgRCcmvo6OGbrvttktj9ZTBYUREkiRJUuW8EZEkSZJUubpIzVpeXnT0wAMPAEXY6dFHH037Pv7xj1d7YA0girqgWPU70tp+9rOfpX2RZjRy5Mg01owtd3siGiSEaF8H8MgjjwDlRgn1FOqsQhSn10rJuuSSS6o+nIYSKQnx3rzgggvSvoULFwJFymq0nAXYc889Adh///3T2EYbbQQUqwO3kjgfedrpO97xDgA+85nP9MsxNYNo0BFpqQBLliwB4OKLL05jkydPBopz3oqiCBqKlrORkvrLX/4y7YuVvfOWqJEaHW1+81XuG93SpUuB4jWS++9//9tu309/+lMA9t57b6C5zkVfyZuQLL/y+X777Ze2I323XhkRkSRJklS5upwCzwu3YlY/2oHutddeaV/cOe+4445p7Etf+hLQuoXs+WzVscceCxSFXltuuWXaZ/SjtjhXALfeeitQRI+i3R3AgQceCMBnP/vZNNZqBezLL0yYt0mtVaSuQhSix0xorfa6UXwe7RUBBg507mhl4rOt1SKUvenrX/86AD/60Y/S2IIFC4Ci0QkUEZNW+cxbmYhKXnnllQCMGTMm7Rs7dmy7x5988skA3H///UC58L2ZF0CM18vDDz+cxvLPOXXsjTfeAIoCfyhaRcdYHhGJaFy9vqb8VpMkSZJUubqfFo8l6eNuL2+hGnnVeX51zERES9FBgwZVcpz1YsKECWk7ZlZj4SWjIO1FTu9hhx0GFFEQKGpqIqc3n/WbP38+UM6hjohIq8prReL9t9tuu5V+gu17oXgtxQKC8XrKRfvZ4cOHp7E8+iv1lah9u/3229NYrfftn//8ZwA+/elPV3NgDSbP4IgFSvOoe2R6xGJ++fv7hhtuAMqR+EYSdUPRqhfgqquuAopIUPyEouY3rud22mmntK8Z2xv3xB//+Eeg/P6M673f/OY3pZ9QZArlEc56an9sRESSJElS5bwRkSRJklS5hsnV+cQnPgGU2/ceffTRQLlF3je/+U0A/vnPfwJw3HHHpX2xqmczy1tW/va3vwVg3333Bcqhus0226zaA6tTUSQYKVl5i2OL5zoWLXojrS0XaVodtfZt5YL2wYMHA0Wx5uGHH572xedZpGuddNJJaV+8p6Uq5E1fajWAiVRoU7Nqy1vQRpvtG2+8MY1Fqvndd98NwF//+te078gjjwTg2muvTWP1WmxcS6RmXXrppWksUoT22GMPoNykI1rixzVMvjxDfD9/4AMf6MMjbhwjRowAYNasWWls+XbledrbhRdeCBSpcfXGiIgkSZKkyjVMRCQMHTo0bU+fPh0oF0PFXeEZZ5wBFIuuQbk1XiOL1p8AH/rQh4Bi9mH27NlpX9z9HnHEEUC5+CvOSz0VLPWHiHrUKkjvSLSVbmUR0Yif+cxXqBUtiTEL2GGttdYCYObMmWns+9//PlDMCD7//PPVH1iDGDVqFFD+XnjmmWeA4rytvfba1R+YmsJ9990HlJvAxIKjIf8u+PznP7/S58yjJDHTH23g77nnnrTvpptuAuCOO+5IY3m71ka0++67A/D6668DRRtaKNqVn3vuuUARIQHYaqutgCKDAcpNPFrNkCFDgHLL/BCfe3/4wx/SWHwG1mvDIiMikiRJkirnjYgkSZKkytVnnKaTYhXTXXbZJY1FitLSpUuBohc3FOlIjZQG8vLLL6ftSB+qlW628847A+Ww70EHHQQUqVkvvvhiu+dt9dSs0JmUrLxo8IknngDgzDPP7NJzNLNaxeeRfjVp0qQ0FgXs+crsedFiq1v+XEQ6g9qL4t1VV101jcVn/+abbw4Ua2Lk8nTeAw88ECi+T9Ta8nTSH//4x0D5PRlrWkRazKabbtrtvyu+ryOlet111233mDlz5qTtek/NeuGFF9L2okWLgPLn/PLy9UHivMd1S57mFmlGebH1qaee2vMDrhORorbxxhunsbvuuguAD3/4w516jmXLlgHF62Xu3Llp3yGHHALUb7MDIyKSJEmSKtdwEZGnnnoqbcfMahSUQTEbFrbddtu0/ZGPfKSPj6735bMJixcvBuCaa65JYxEJqeWKK64o/Xm//fZL2+uvv35vHWJduvfeewHYYYcdevxcEQmJImIoZilauQVtZ0T0MS9Mj/dtrUK7ZnH++een7Sio7ui9mps8eXLpz/ksmWrLX1/Tpk0DihnZ+Jn79re/nbajYHjKlClAY35P9JVjjz22vw+hcvk1RK1I7ciRI4GiuLo3Zplfe+21Hj9HPYhlFgDGjRsHwIknntil54hmE3nmR+hMM4BGFN8N22+/fRqLDIto7QzFtd8rr7zS7jn+8Y9/APDAAw8AcN5556V9EWWqV0ZEJEmSJFWu7iMicXcci6DlOYJPPvnkCn8vakXyFm+1FmSqd6eddlrajgWO9tlnnxU+Pl8EKNrfxYzqOeeck/blOdXNaMcddwSKBaOgWBiuoyjJWWedlbYnTpxY2pfnlZ999tmAdSErE7Nad955Zz8fSbXymeRYPPS2225LY8svzPXSSy+l7VtuuaW0L1p7asUilx+K2cWoEclbfsYCfDFrCHD99dcDsM022wBwwgkn9O3BNoCrr74aKL/2YkHcaDkN5QWDm0U+e3zdddcB5frKeO1EBC2vf+tqdCRqKsaMGdNuX3y3fOMb3+jSc/anxx9/PG3H+y7/bAvRRjbqGqDIYoiarbj2AzjqqKOA4j3abDbccEOgXIeaL9TdGdHCPK6VY3FvqP/rPSMikiRJkirnjYgkSZKkytVVala0lI0QMBSpSXnIryOjR48GihSbKBRtVLXCa3mqwaxZs0qPz8OZEeKcOnUqULs1YLOKYvIoRIUiTShPN4h0mWjHm4u0rmiR3MppWLFqep5iFakJtdphx7nuqHVjXmDcbPJi6Egb+uAHP5jGttxyS6BYBXf+/PlpX6SQRsH7sGHD+vZgm0Cedjt27NjSvi222CJtH3DAAQB88pOfTGOPPfYYUKTEHX/88WnfwIGtOVd38MEHA7XTmSPFFYq0w2aSFwzffPPNQNE6H4o0rWjdHilFUHxObrDBBu2ed8GCBe3GooHMo48+2m7fjBkzANhuu+269g/oR3maWpyfNddcs93joqV2fP5Bsdp6yFsV/+AHPwCKlPtmc/nllwMrb66z3nrrAUVDjbw8Ia59G7ENeWt+ykqSJEnqV/0WEVmyZEnaXrhwIVDM4D/88MOdeo4o8MoXtol2vY1YmL4yMdOXz/hddNFF/XU4dW3mzJkAnH766WksoiOXXXZZGosC9Jh1zlscRwFZq4rZPSgv8hWiDW9XRdveZm59nDc9iALWBx98MI3dc889QBEZGT9+fNq3ySabAM3bqrI/DRo0CCj//+y///5A8X/S7Itr/vvf/wZg+vTpQFGsD0WzkzgH+fdoLD6Xt8RvdjFDnTeQiOuOaKEaC89BESGqFSmqFfUI8RkRCxtCOTLTKPLmOtHKNy+2j2jS008/DZSzNOL7Np5j7733TvuaNRISVl99daD292xH8mhvIzMiIkmSJKly3ohIkiRJqlwlqVmvvvpq2o5+0PlqkVEs2JFIU4iiJShWOF1llVV65TjVPCKkPmfOnH4+ksbV1TBxR6K3OTR3SlbICzSjWcRbb72VxmL9gEgVirQXVWPPPfdM2yNGjAA6nxLciBYvXpy2f/WrXwFwyimnAOW1HOIcREpWrMUCxXdvrNHUSvI0qSgQfuihhwDYfffd2z2+ozSsXBQYR+OPRk97y1Oo9tprL6C8Qnr+OoSi+BrK69OotRgRkSRJklS5PomIzJs3Dyju8u+44460L29TuSL56qRRbByzqM4cStXIoxi1oiOxP1r65u14WyHq0VX5bOE666zTj0eifLXn5557rh+PpG8tWrQIKH8Hx6rWkUmQt009/PDDgeK9nM/2r7baan17sA0iop277rorUG5Be8UVVwBw1VVXAeVi/1ix/dBDD01jcU6bOasjj3rk21IwIiJJkiSpcn0SEYkc1GnTpq3wMVtvvXXajkWmYqbmW9/6VtrXiIuzSM0gj2p0FOEw+qFGEzPWUETpo91oM7V+Hzp0KADjxo1rt8/3bc/E6yRf6HbChAmln5JWzoiIJEmSpMp5IyJJkiSpcn2SmnXMMceUfkqSVC9qtaCNNssDBzo/J0lV8RNXkiRJUuUqWdBQkqR6MWrUqLSdLzQpSaqWERFJkiRJlfNGRJIkSVLlvBGRJEmSVDlvRCRJkiRVbkBbW1vnHzxgwDPA/L47nIYxrK2tbUhXfsFzl3T53IHnL+Nrr/s8dz3j+es+z13PeP66z3PXfV6v9Eynzl+XbkQkSZIkqTeYmiVJkiSpct6ISJIkSaqcNyKSJEmSKueNiCRJkqTKeSMiSZIkqXLeiEiSJEmqnDcikiRJkirnjYgkSZKkynkjIkmSJKly/weGaP1Pvq9h9wAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"fix, ax = plt.subplots(nrows=1, ncols=10, \n",
" sharex=True, sharey=True, \n",
" figsize=(14, 8))\n",
"ax = ax.flatten()\n",
"for i in range(10):\n",
" img = X_train[y_train == 5][i].reshape(28, 28)\n",
" ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
"ax[0].set_xticks([])\n",
"ax[0].set_yticks([])\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***\n",
" \n",
"## 3. Building a Convolutional Neural Network using TensorFlow"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Introducing CNNs:**\n",
"\n",
"Convolutional Neural Networks (CNNs) employ feature heirarchy: the early layers at the input extract low-level features (e.g. edges, corners), while the later layers combine these to form higher-level features (e.g. shapes, objects). A typical CNN consists of convolutional (conv) layers, subsampling through pooling layers, and one or more fully-connected layers (e.g. a multilayer perceptron) at the end. Both the convolutional and fully-connected layers have weight parameters and biases that are 'learned' as we train our CNN. \n",
"\n",
"An excellent introductory video explaining the basics of CNNs can be found at this link . Roughly speaking, we can think of the convolution operations as sliding a 'filter' across the layer input. These filters specialize in detecting different features. Some might produce a large output value when they slide over corners; others might produce a large output when they slide over edges of a specific orientation, etc. Usually each convolution layer has several output channels, each of which come to specialize in detecting different low-level features. \n",
"\n",
"The CNN architecture we will use is illustrated in the diagram below and is relative simply. For a primer on more advanced CNN architectures including many important recent developments, I recommend reading the papers mentioned here .\n",
"\n",
" \n",
"\n",
"Our simple CNN will consist of two convolutional layers (using 5x5-pixel kernels and \"valid\"-type padding where we drop the perimeter pixels) each followed by max-pooling layers having 2x2 downsampling. Next we'll flatten the output channels into vectors and feed these to a fully connected layer (fc_3) having a Rectified Linear Unit (ReLU) activation function, followed by a second fully-connected layer (fc_4) to whose output we'll apply the softmax function to determine the most probable class label (i.e. which digit between 0 and 9). Between layers fc_3 and fc_4, we'll apply a popular regularization technique called 'dropout' to help prevent overfitting. A fraction of our hidden units (optimally 50%) will be randomly dropped with each training iteration, which forces our network to learn more general and robust patterns from the training data. \n",
"\n",
"Our CNN has several hyperparameters, including:\n",
"* learning rate\n",
"* convolutional kernel size and padding type\n",
"* max-pooling strides\n",
"* number of output channels after each conv layer\n",
"* number of units in the fc_3 layer\n",
"\n",
"Some of these we'll hard-code, others we'll set as variables. Note that the dropout rate is also technically a hyperparameter, but in this case it's already understood to be optimal at 50%. \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Batch Generator:**\n",
"\n",
"Let's first define a helper function that splits our total training set into mini-batches that we train our CNN on sequentially. There are two main reasons for doing this:\n",
"\n",
"* The use of mini-batches injects some noise into our gradient-based updates to model parameters (e.g. the weights between nodes in our network), helping us escape local minima and thereby optimize to a better-performing model. \n",
"\n",
"* GPUs generally have less available RAM than CPUs, which can be a problem for large datasets. \n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"# define a function for iterating through mini-batches of the data\n",
"def batch_generator(X, y, batch_size=100, \n",
" shuffle=False, random_seed=None):\n",
" ind = np.arange(y.shape[0]) # create indices\n",
"\n",
" if shuffle: # shuffles sample order in output batch\n",
" rng = np.random.RandomState(random_seed) # random num gen\n",
" rng.shuffle(ind) # use random nums to shuffle indices\n",
" X = X[ind]\n",
" y = y[ind]\n",
"\n",
" for i in range (0, X.shape[0], batch_size):\n",
" yield (X[i:i+batch_size, :], y[i:i+batch_size])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Implementing the CNN in Low-Level TensorFlow**:\n",
"\n",
"TensorFlow provides an interface between easy-to-use Python and the more tempermental and tricky C++ that is used to communicate with the GPU.\n",
"\n",
"We'll define our CNN as a class, using TensorFlow's low-level API. The core idea is that we first define a computational graph that gets compiled and run during a TensorFlow session. This graph consists of all tensors, variables, layers, and arithmetic involved in our model definition. To feed data into our network, we must pre-define data **placeholders**, which we feed with dictionaries containing our data during training. Prior to running any calculations, all variables in our computational graph must first be initialized. \n",
"\n",
"The following commented code walks you through the process of building a model object using TensorFlow, including methods for saving and loading the model parameters. \n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"class DigitCNN(object):\n",
" def __init__(self, batchsize=100, \n",
" epochs=10, learning_rate=1e-4,\n",
" conv_1_output_channels=32, \n",
" conv_2_output_channels = 64, \n",
" dropout=0.5, fc_3_output_units=1024, \n",
" shuffle=True, random_seed=None):\n",
" \n",
" np.random.seed(random_seed) \n",
" # seeds np for reproducability \n",
" # between identical training instances\n",
" \n",
" self.batchsize = batchsize \n",
" # how many training samples we feed to \n",
" # our GPU at a time\n",
" \n",
" self.epochs = epochs \n",
" # number of training epochs\n",
" \n",
" self.learning_rate = learning_rate \n",
" # scales the weight updates\n",
" \n",
" self.conv_1_output_channels = conv_1_output_channels \n",
" # num output channels from Conv_1 layer\n",
" \n",
" self.conv_2_output_channels = conv_2_output_channels \n",
" # num output channels from Conv_2 layer\n",
" \n",
" self.fc_3_output_units = fc_3_output_units \n",
" # num output units from Fully-Connected layer 3\n",
" \n",
" self.dropout = dropout \n",
" # dropout probability for neurons in our FC layers\n",
" \n",
" self.shuffle = shuffle \n",
" # if True, shuffles the training samples between epochs\n",
" \n",
" self.history = pd.DataFrame(\n",
" columns=['Epoch','Loss','Accuracy']) \n",
" # for tracking performance\n",
" \n",
" # TensorFlow needs to construct the computational graph \n",
" # before we begin feeding it data:\n",
" self.g = tf.Graph()\n",
" with self.g.as_default():\n",
"\n",
" # set TensorFlow's random seed to match that of numpy\n",
" tf.set_random_seed(random_seed)\n",
"\n",
" # construct the graph for our CNN\n",
" self.build()\n",
"\n",
" # initialize the values of any variables \n",
" # we have explicitly defined in our graph\n",
" self.init_op = tf.global_variables_initializer()\n",
"\n",
" # TensorFlow's saver; will be used to \n",
" # save the CNN variables after training\n",
" self.saver = tf.train.Saver()\n",
"\n",
" # create a TensorFlow session using \n",
" # our computational graph\n",
" self.sess = tf.Session(graph=self.g)\n",
"\n",
" # *** defines the computational graph built for our CNN ***\n",
" def build(self):\n",
"\n",
" # define placeholders for X (images) and y (labels), \n",
" # which we will feed data into later\n",
" tf_x = tf.placeholder(\n",
" dtype=tf.float32, \n",
" shape=[None, 28*28], \n",
" name='tf_x')\n",
" \n",
" tf_y = tf.placeholder(\n",
" dtype=tf.int32, \n",
" shape=[None], \n",
" name='tf_y')\n",
"\n",
" # since we feed X in as batches, we must \n",
" # reshape it into a Rank 4 tensor,\n",
" # tensor dimensions: [batchsize, width, height, 1]\n",
" # notes: \n",
" #- the last digit specifies the number of \n",
" # channels (3 for RGB, 1 for greyscale)\n",
" # - 28x28 comes from the number of pixels\n",
" # - specifying -1 for a dimension tells TF \n",
" # to compute it based on existing constraints\n",
" tf_x_image = tf.reshape(\n",
" tf_x, \n",
" shape=[-1, 28, 28, 1], \n",
" name='input_x_2dimages')\n",
"\n",
" # one-hot encoding assigns labels 0-9 \n",
" # their own basis vectors, e.g. [1, 0, 0 ...]\n",
" tf_y_onehot = tf.one_hot(\n",
" indices=tf_y, \n",
" depth=10, \n",
" dtype=tf.float32, \n",
" name='input_y_onehot')\n",
"\n",
" # ** We begin building up our convolutional neural network **\n",
" \n",
" # first layer: Conv_1 (convolutional)\n",
" print('\\nBuilding Conv_1 Layer:')\n",
" h1 = self.conv_layer(\n",
" tf_x_image, name='conv_1', \n",
" kernel_size=(5, 5), \n",
" padding='VALID', \n",
" n_output_channels=self.conv_1_output_channels)\n",
" \n",
" # MaxPooling of Conv_1\n",
" h1_pool = tf.nn.max_pool(\n",
" h1, \n",
" ksize=[1, 2, 2, 1], \n",
" strides=[1, 2, 2, 1], \n",
" padding='SAME')\n",
"\n",
" # second layer: Conv_2 (convolutional)\n",
" print('\\nBuilding Conv_2 Layer:')\n",
" h2 = self.conv_layer(\n",
" h1_pool, name='conv_2', \n",
" kernel_size=(5, 5), \n",
" padding='VALID', \n",
" n_output_channels=self.conv_2_output_channels)\n",
" \n",
" # MaxPooling of Conv_2\n",
" h2_pool = tf.nn.max_pool(\n",
" h2, \n",
" ksize=[1, 2, 2, 1], \n",
" strides=[1, 2, 2, 1], \n",
" padding='SAME')\n",
"\n",
" # third layer: fc_3 (fully-connected)\n",
" print('\\nBuilding fc_3 layer:')\n",
" h3 = self.fc_layer(\n",
" h2_pool, name='fc_3', \n",
" n_output_units=self.fc_3_output_units, \n",
" activation_fn=tf.nn.relu) \n",
" # note: relu = rectified linear unit\n",
"\n",
" # neuron dropout for fc_3\n",
" # (we will feed this a value)\n",
" keep_prob = tf.placeholder(\n",
" tf.float32, \n",
" name='fc_keep_prob') \n",
" \n",
" h3_drop = tf.nn.dropout(\n",
" h3, keep_prob=keep_prob, \n",
" name='dropout_layer')\n",
"\n",
" # fourth layer: fc_4 (fully-connected)\n",
" print('\\nBuilding 4th layer:')\n",
" h4 = self.fc_layer(\n",
" h3_drop, name='fc_4', \n",
" n_output_units=10, # matches num. unique labels\n",
" activation_fn=None) # we use linear activation here\n",
"\n",
" # ** now we add to our computational graph \n",
" # all the other functions needed to perform \n",
" # training, predictions, and validation **\n",
" \n",
" # generate probabilities with softmax, \n",
" # and take the logit with largest activation \n",
" # as the class\n",
" predictions = {\n",
" 'probabilities': tf.nn.softmax(h4, \n",
" name='probabilities'),\n",
" 'labels': tf.cast(tf.argmax(h4, axis=1), \n",
" tf.int32, name='labels')}\n",
" \n",
" # we will use softmax cross entropy \n",
" # as our loss function (to be minimized)\n",
" cross_entropy_loss = tf.reduce_mean(\n",
" tf.nn.softmax_cross_entropy_with_logits_v2(\n",
" logits=h4, labels=tf_y_onehot),\n",
" name='cross_entropy_loss')\n",
"\n",
" # we'll perform optimization using the \n",
" # AdamOptimizer, a robust and popular \n",
" # gradient-based method\n",
" optimizer = tf.train.AdamOptimizer(self.learning_rate)\n",
" optimizer = optimizer.minimize(\n",
" cross_entropy_loss, \n",
" name='train_op')\n",
"\n",
" # getting the number of correct \n",
" # predictions and corresponding accuracy\n",
" correct_predictions = tf.equal(\n",
" predictions['labels'], \n",
" tf_y, name='correct_preds')\n",
" \n",
" accuracy = tf.reduce_mean(\n",
" tf.cast(correct_predictions, tf.float32), \n",
" name='accuracy')\n",
"\n",
" # *** create Wrapper for convolution layers ***\n",
" def conv_layer(self, input_tensor, name,\n",
" kernel_size, n_output_channels,\n",
" padding='SAME', strides=(1, 1, 1, 1)):\n",
"\n",
" with tf.variable_scope(name):\n",
" # note: input tensor shape is [batchsize, \n",
" # width, height, input_channels]\n",
" input_shape = input_tensor.get_shape().as_list()\n",
" n_input_channels = input_shape[-1]\n",
" weights_shape = list(kernel_size) \\\n",
" + [n_input_channels, n_output_channels]\n",
" weights = tf.get_variable(\n",
" name='_weights', \n",
" shape=weights_shape)\n",
" print(weights)\n",
" biases = tf.get_variable(\n",
" name='_biases', \n",
" initializer=tf.zeros(shape=[n_output_channels]))\n",
" print(biases)\n",
" conv = tf.nn.conv2d(\n",
" input=input_tensor, \n",
" filter=weights, \n",
" strides=strides, \n",
" padding=padding)\n",
" print(conv)\n",
" conv = tf.nn.bias_add(\n",
" conv, \n",
" biases, \n",
" name='net_pre-activation')\n",
" print(conv)\n",
" conv = tf.nn.relu(conv, name='activation')\n",
" print(conv)\n",
"\n",
" return conv\n",
" \n",
" # *** create wrapper for fully-connected layers ***\n",
" def fc_layer(self, input_tensor, name,\n",
" n_output_units, activation_fn=None):\n",
"\n",
" with tf.variable_scope(name):\n",
" input_shape = input_tensor.get_shape().as_list()[1:] \n",
" # everything but batch size\n",
" n_input_units = np.prod(input_shape)\n",
"\n",
" if len(input_shape) > 1:\n",
" input_tensor = tf.reshape(\n",
" input_tensor, \n",
" shape=(-1, n_input_units))\n",
"\n",
" weights_shape = [n_input_units, n_output_units]\n",
" weights = tf.get_variable(\n",
" name='_weights', \n",
" shape=weights_shape)\n",
" print(weights)\n",
" biases = tf.get_variable(\n",
" name='_biases', \n",
" initializer=tf.zeros(shape=[n_output_units]))\n",
" print(biases)\n",
" layer = tf.matmul(input_tensor, weights)\n",
" print(layer)\n",
" layer = tf.nn.bias_add(\n",
" layer, \n",
" biases, \n",
" name='net_pre-activation')\n",
" print(layer)\n",
"\n",
" if activation_fn is None:\n",
" return layer\n",
"\n",
" layer = activation_fn(layer, name='activation')\n",
" print(layer)\n",
"\n",
" return layer\n",
" \n",
" # *** saving the model ***\n",
" def save(self, epoch, path='./digitCNN-model/'):\n",
" if not os.path.isdir(path):\n",
" os.makedirs(path)\n",
" print('Saving model in %s' % path)\n",
" self.saver.save(\n",
" self.sess, \n",
" os.path.join(path, 'model.ckpt'), \n",
" global_step=epoch)\n",
"\n",
" # *** loading the model from saved data ***\n",
" def load(self, epoch, path):\n",
" print('Loading model from %s' % path)\n",
" self.saver.restore(\n",
" self.sess, \n",
" os.path.join(path, 'model.ckpt-%d' % epoch))\n",
"\n",
" # *** training the CNN ***\n",
" def train(self, training_set, \n",
" validation_set=None, initialize=True):\n",
"\n",
" # initialize variables\n",
" if initialize:\n",
" self.sess.run(self.init_op)\n",
"\n",
" X_data = np.array(training_set[0])\n",
" y_data = np.array(training_set[1])\n",
" \n",
" epoch0 = self.history['Epoch'].shape[0] \n",
" # how many epochs have already been trained\n",
" \n",
" for epoch in range(1, self.epochs + 1):\n",
" batch_gen = batch_generator(X_data, \n",
" y_data, \n",
" shuffle=self.shuffle)\n",
"\n",
" avg_loss = 0.0\n",
" for i, (batch_x,batch_y) in enumerate(batch_gen):\n",
" feed = {'tf_x:0': batch_x,\n",
" 'tf_y:0': batch_y,\n",
" 'fc_keep_prob:0': self.dropout} # for dropout\n",
" loss, _ = self.sess.run(\n",
" ['cross_entropy_loss:0', 'train_op'], \n",
" feed_dict=feed)\n",
" avg_loss += loss\n",
"\n",
" print('Epoch %02d: Training Avg. Loss: %7.3f' \\\n",
" % (epoch, avg_loss), end=' ')\n",
"\n",
" # check accuracy on validation set if supplied\n",
" if validation_set is not None:\n",
" feed = {'tf_x:0': validation_set[0],\n",
" 'tf_y:0': validation_set[1],\n",
" 'fc_keep_prob:0': 1.0} \n",
" # we DON'T want neuron dropout when making predictions\n",
" valid_acc = self.sess.run('accuracy:0', feed_dict=feed)\n",
" print(', Validation Acc: %7.3f' % valid_acc)\n",
" # update our CNN's performance history\n",
" self.update_history((epoch0 + epoch, \n",
" avg_loss, \n",
" valid_acc)) \n",
" else:\n",
" self.update_history((epoch0 + epoch, avg_loss))\n",
" print() \n",
"\n",
" # *** making predictions with our trained CNN ***\n",
" def predict(self, X_test, return_proba=False):\n",
" feed = {'tf_x:0' : X_test,\n",
" 'fc_keep_prob:0': 1.0} \n",
" # we DON'T want neuron dropout when making predictions\n",
"\n",
" if return_proba:\n",
" return self.sess.run('probabilities:0', \n",
" feed_dict=feed)\n",
" else:\n",
" return self.sess.run('labels:0', \n",
" feed_dict=feed)\n",
" \n",
" # *** exporting log files that allow us to \n",
" # visualize our graph using TensorBoard ***\n",
" def export_logs(self):\n",
" self.sess.run(self.init_op)\n",
" file_writer = tf.summary.FileWriter(\n",
" logdir='./logs/', \n",
" graph=self.g)\n",
" \n",
" # *** updating our CNN's performance history ****\n",
" def update_history(self, entry):\n",
" # note: entry is a typle of form \n",
" # (epoch_number, avg_loss, valid_acc)\n",
" if len(entry) is 3: \n",
" # case where we provided a \n",
" # validation set to compute accuracy\n",
" new_entry = pd.DataFrame(\n",
" columns=['Epoch', 'Loss', 'Accuracy'])\n",
" new_entry.loc[0] = [entry[0], entry[1], entry[2]]\n",
" else:\n",
" new_entry = pd.DataFrame(columns=['Epoch', 'Loss'])\n",
" new_entry.loc[0] = [entry[0], entry[1]]\n",
" \n",
" # refresh the dataframe row indexing\n",
" self.history = self.history.append(new_entry)\n",
" self.history = self.history.reset_index(drop=True)\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that I have chosen to stick with TensorFlow's low-level API. Some of the above steps can be simplified by making use of its higher-level **Layers** API. \n",
"\n",
"Let's now initialize an instance of our newly-defined CNN class. We'll set its training to 30 epochs. Creating our CNN object and building the computational graph:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Building Conv_1 Layer:\n",
"\n",
"\n",
"Tensor(\"conv_1/Conv2D:0\", shape=(?, 24, 24, 32), dtype=float32)\n",
"Tensor(\"conv_1/net_pre-activation:0\", shape=(?, 24, 24, 32), dtype=float32)\n",
"Tensor(\"conv_1/activation:0\", shape=(?, 24, 24, 32), dtype=float32)\n",
"\n",
"Building Conv_2 Layer:\n",
"\n",
"\n",
"Tensor(\"conv_2/Conv2D:0\", shape=(?, 8, 8, 64), dtype=float32)\n",
"Tensor(\"conv_2/net_pre-activation:0\", shape=(?, 8, 8, 64), dtype=float32)\n",
"Tensor(\"conv_2/activation:0\", shape=(?, 8, 8, 64), dtype=float32)\n",
"\n",
"Building fc_3 layer:\n",
"\n",
"\n",
"Tensor(\"fc_3/MatMul:0\", shape=(?, 1024), dtype=float32)\n",
"Tensor(\"fc_3/net_pre-activation:0\", shape=(?, 1024), dtype=float32)\n",
"Tensor(\"fc_3/activation:0\", shape=(?, 1024), dtype=float32)\n",
"\n",
"Building 4th layer:\n",
"\n",
"\n",
"Tensor(\"fc_4/MatMul:0\", shape=(?, 10), dtype=float32)\n",
"Tensor(\"fc_4/net_pre-activation:0\", shape=(?, 10), dtype=float32)\n"
]
}
],
"source": [
"cnn = DigitCNN(epochs=30, random_seed=27)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"TensorFlow has a handy tool called ***TensorBoard*** which can be used to visualize computational graphs. First we must export the graphs as follows:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"# create graph log to visualize with TensorBoard\n",
"cnn.export_logs()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Instructions on how to use TensorBoard can be found here . Below we view the computational nodes connected to each of our layers:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also expand each layer (in this case fc_3) to see a more detailed breakdown of the computational nodes within:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***\n",
" \n",
"## 4. Training our CNN"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Before training our CNN, we must standardize our data. We'll also create a validation set using the last 10000 samples of our 60000-sample training set. (Note that we would typically use k-fold cross validation.)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"# we then need to normalize the data (mean centering \n",
"# and division by the standard deviation) for better \n",
"# training performance and convergence\n",
"mean_vals = np.mean(X_train, axis=0)\n",
"std_val = np.std(X_train)\n",
"\n",
"# obtain the standardized version of our data:\n",
"X_train_standardized = (X_train - mean_vals)/std_val\n",
"X_test_standardized = (X_test - mean_vals)/std_val\n",
"\n",
"# instead of k-fold cross-validation, we'll just \n",
"# use the last 10000 training entries\n",
"X_train_standardized_subset, y_train_subset = \\\n",
" X_train_standardized[:50000,:], y_train[:50000]\n",
"X_valid_standardized, y_valid = \\\n",
" X_train_standardized[50000:,:], y_train[50000:]\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we train our CNN and save the fitted model. "
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 01: Training Avg. Loss: 200.779 , Validation Acc: 0.971\n",
"Epoch 02: Training Avg. Loss: 53.082 , Validation Acc: 0.983\n",
"Epoch 03: Training Avg. Loss: 36.265 , Validation Acc: 0.983\n",
"Epoch 04: Training Avg. Loss: 28.832 , Validation Acc: 0.987\n",
"Epoch 05: Training Avg. Loss: 23.475 , Validation Acc: 0.989\n",
"Epoch 06: Training Avg. Loss: 20.583 , Validation Acc: 0.989\n",
"Epoch 07: Training Avg. Loss: 17.263 , Validation Acc: 0.990\n",
"Epoch 08: Training Avg. Loss: 14.779 , Validation Acc: 0.990\n",
"Epoch 09: Training Avg. Loss: 13.383 , Validation Acc: 0.991\n",
"Epoch 10: Training Avg. Loss: 11.807 , Validation Acc: 0.991\n",
"Epoch 11: Training Avg. Loss: 10.714 , Validation Acc: 0.991\n",
"Epoch 12: Training Avg. Loss: 9.549 , Validation Acc: 0.992\n",
"Epoch 13: Training Avg. Loss: 8.330 , Validation Acc: 0.992\n",
"Epoch 14: Training Avg. Loss: 7.352 , Validation Acc: 0.991\n",
"Epoch 15: Training Avg. Loss: 6.533 , Validation Acc: 0.991\n",
"Epoch 16: Training Avg. Loss: 5.983 , Validation Acc: 0.991\n",
"Epoch 17: Training Avg. Loss: 5.140 , Validation Acc: 0.992\n",
"Epoch 18: Training Avg. Loss: 5.058 , Validation Acc: 0.992\n",
"Epoch 19: Training Avg. Loss: 4.327 , Validation Acc: 0.993\n",
"Epoch 20: Training Avg. Loss: 4.234 , Validation Acc: 0.993\n",
"Epoch 21: Training Avg. Loss: 3.883 , Validation Acc: 0.992\n",
"Epoch 22: Training Avg. Loss: 3.291 , Validation Acc: 0.992\n",
"Epoch 23: Training Avg. Loss: 2.970 , Validation Acc: 0.991\n",
"Epoch 24: Training Avg. Loss: 2.807 , Validation Acc: 0.991\n",
"Epoch 25: Training Avg. Loss: 2.649 , Validation Acc: 0.993\n",
"Epoch 26: Training Avg. Loss: 2.484 , Validation Acc: 0.991\n",
"Epoch 27: Training Avg. Loss: 2.275 , Validation Acc: 0.992\n",
"Epoch 28: Training Avg. Loss: 2.320 , Validation Acc: 0.992\n",
"Epoch 29: Training Avg. Loss: 2.375 , Validation Acc: 0.993\n",
"Epoch 30: Training Avg. Loss: 2.270 , Validation Acc: 0.992\n",
"Saving model in ./digitCNN-model/\n"
]
}
],
"source": [
"# train our neural network\n",
"cnn.train(training_set=(X_train_standardized_subset, \n",
" y_train_subset),\n",
" validation_set=(X_valid_standardized, \n",
" y_valid),\n",
" initialize=True)\n",
"\n",
"# save the model after training \n",
"cnn.save(epoch=30)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's print the loss function (cross-entropy) and training accuracy, as a function of the number of training epochs:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA0sAAAGbCAYAAAAV5QtwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzs3Xd4lFX6xvHvQ28WisSOIDYsuAqsXdaGrmJZbKy9YS8/6yprd0VZ6wIW7IKCa++CiGJFARUXRERBBZQO0kvI+f3xzJghmSRvksm8k+T+XFeuSd56JiC+95xznmMhBERERERERGRddeJugIiIiIiISC5SWBIREREREUlDYUlERERERCQNhSUREREREZE0FJZERERERETSUFgSERERERFJQ2FJREREREQkDYUlERERERGRNBSWRERERERE0qgXdwPiVqdOndC4ceO4myEiIiIiUqLly5eHEII6OrKs1oelxo0bs2zZsribISIiIiJSIjNbEXcbaiOlUxERERERkTQUlkRERERERNJQWBIREREREUlDYUlERERERCQNhSUREREREZE0FJZERERERETSUFgSERERERFJQ2FJREREREQkDYUlERERERGRNBSWRERERERE0shuWDI7FrMXMfsZsxWYTcasD2brFTmuOWaPYjYPs2WYjcBs5zTXa4TZvzH7LXG9zzDbL1tvR0REREREaq5s9yxdCawFrgMOBR4EzgfexczbYmbAa4n9FwM9gPrA+5htXuR6jwHnADcARwC/AcMw27XK34mIiIiIiNRoFkLI4t1sI0KYW2TbqcBTwIGEMBKzo4BXgAMI4f3EMRsA04DBhHBJYltH4GvgTEJ4IrGtHjARmEwIR0ZpUtOmTcOyZcsq/95ERERERKqImS0PITSNux21TXZ7looGJTcm8bpZ4vVI4Nc/gpKf9zvwOnBUynlHAmuA51KOyweGAt0wa5ixdleFvfaCG2+MuxUiIiISk/feg7/+FWbPjrslIlKSXCjwsH/idVLidUdgQprjJgJbYtYs5bhphLA8zXENgPaZbmhGzZkDU6bE3QoRERGJwW+/wYknwttvw9//DmvXxt0iEUkn3rBkthlwCzCCEMYmtrYAFqY5ekHitXnE41qUct9emI3FbCz5+eVudkbk5emjJBERkVqooABOPRWWLYPrroORI+Hmm+NulYikUy+2O3sP0atAPnBG6h4g3UQqS/NzlOOKC2EgMBCApk2zOGkrxcYbw3ffxXJrERERic/dd8OIETBwIJxzDvz6K9x2G+y9N3TrFnfrRCRVPD1LZo3winftgG6EMCNl7wLS9wole5QWRjxuQZp9uUM9SyIiIrXO2LHem9SjB5x9tm8bMAB22glOPhlmzCj9fBHJruyHJbP6wItAF+CvhPC/IkdMxOcjFdUB+IUQlqYc1xazJmmOWw38kLlGV4G8PJg/H9asibslIiIikgVLlkDPnrDJJvDII2CJsTBNmsDzz8PKlXDCCXo0EMkl2R2G52spPQMcCBxOCKPTHPUacAZm+xPCqMR56wPdgWeLHHczcBxeejxZOvwEYDghrKqid5EZeXn+OncubLppvG0REZGsCsEflvv2hQYNoHlzaNHCv0r6PvnzhhtCvfgG0UslXHwxTJ0KH3zgf5apttsOHn3Uiz784x8+VE9E4pftf24H4OHmX8AyzPZI2TcjMRzvNeAzYDBmV+HD7q7F5yL1/ePoEL7G7DngvkRv1TR8gdu2wElZeC+VkwxLs2crLImI1CK//ebDr956y1eR2GwzWLAAZs6ECRP8+8WLS7/GBhuUHrD+8hfo1Ck770eiGTIEnnoKbrgB9t03/TEnnAAffQT33AP77APHHJPdNopIcdlelPYnoE0Je28mhJsSx7UA7gKOBhrh4elyQhhf5HqN8eD1d2BDYDxwDSF8ELVJsS1K++mnPpPz7bfh0EOzf38REcm6F1+Ec8/1Kmh9+8KFF0KdNAPi16yBRYtg4UIPT8mv1J9L2pef78O7evWCPn2K92BI9k2bBrvuCjvv7L1KpfUMrlrlYWryZPjyS9h666w1U3KcFqWNR3bDUg6KLSz9+CO0bw9PPgmnnZb9+4uISNb8/rsPwRo0CHbfHQYPhu23z/x9QvDA9K9/wf33Q6tWcN99PrTLyq4VK1VgzRrYbz+YNAnGj4c2JX1knOKnn2C33WCrrfyz1UaNqrqVNUNBgS/0+8ADMHEiHHGE/93v3Llm/P1XWIpHLixKWzulDsMTEZEaa+RI71F49lm48Ub47LOqCUrgD4QtW/owrjFjYMstfcHTbt38MzrJvptugtGjfY5alKAEHpKefhq++gouu6wqW1czLFwI997r/10dcgh8/DG0bQv9+8Of/+yfTffu7cNcRcpLYSkuzZp5+RuFJRGRGmnFCvi//4MDD4TGjb2H4KaboH797Nx/t938Ib1fP3/daSfvcVq9Ojv3F3j/fR8KedZZcNxx5Tv3iCPgmmvg4YfhmWeqpn3V3Zdf+vy/zTaDyy/3ntRBg7z8+rBh/oj12GM+lPGOO/xDi5139v8O9OGBRKVheHENwwNo185n9w4eHM/9RUSkSnz5pa+ZM2kSXHQR3Hmnfz4Wl5kzvYfihRdghx3goYd8aJhUnfnzoWNH/2x03DhoWoHBU/n5cMABfv6YMdChQ+bbWd2sXOll1gcMgM8/9/+uTjoJzj8f/vSnks+bPdv//g8ZAp984ts6d/Zheiec4IEr12kYXjzUsxQnLUwrIlKj5OfDbbf50J/ff/dPt/v1izcogT8IPv88vPEGLF8O++/vvR3z58fbrpoqBP/9zp3rD+cVCUrghSCGDvXAdeyxsHRp2efUVNOmeU/b5pvDqaf60Lv77vMPAgYOLD0ogT9yXXihD9H7+WcvsLJ2LVxxBWyxBXTt6h8izJuXlbcj1YjCUpwUlkREaowpU7yK2fXX+4Pt//7n8ydyyeGH+8T3q6/2Mtbbb++vtXyQScY99BC8+qoP/SrrIb4sm27q892++w7OO692/VmtXesl9o84wofS3X23B/0RI/z3cemlvu5YeW25JVx1lffYffedD4+dPdt7pzbeGA47zOeMlVXCX2oHDcOLcxjeuefCK68oMImIVGMh+MPxlVf6ArMPPuhDe3LdN9/4/4ZGjy78VH277eJuVfU3YYIP7+raFd58M31p+Iq49VZfo+mhh/zPrSabNw8ef9zf67Rp/tlyr17+tfnmVXPPEPy/iSFDvDfv55+hYUP/gOHEEz2wNW5cNfeOSsPw4qGwFGdYuuGGwtm2devG0wYREamwX3/14VbvvOO9SI8/Xj3mPiQVFHiVtn/8w4fnXXutf1/VpaqXLfNKb+PG+RCo7t2zV/iiKq1YAV26+PC78eMLC99mQkEB/PWvXjTis8+8gEdNEgJ88YWX/X7uOV9var/9fOjc0Uf7BxHZbMvo0R6annvOP9Nu1gyOOsp79/bZJ3ttSaWwFA+FpTjD0oABPvN31qzM/osqIiJV7r//9WE7K1bAXXf599V1LZfZs72a2LPPwjbb+Cf6BxyQmWuvXu1DEseMKfyaONEf/pM23dR7Dc45x7+vri66yP/X/s47Xq490+bO9WF9DRt60KzIELTKWLMm88MAV670wgsDBnhhlGbNfE7S+ed7Bce4rV0Lo0Z5j9OLL3oJ8iuuiKctCkvxUFiKMyy98ILXEh0/HnbZJZ42iIhIuSxc6A/Fzz7rvQhPP11zhq8NHw4XXOBllU8+2eeItG4d/fyCApg8uTAUffGF/y9u1Srf37KlD1FLfu2+uz8gJwNG3bpwzDHehq5dq1f4fO0173m4/HL/vVWVTz/1eTtHHAEvvZSd39HHH3vhkmHDqu4eO+7of+4nnwzrr19196mM1as9MFa0YEdlKSzFQ2EpzrD00Ufexzx8OBx8cDxtEBGpIQoK/EHy3Xd9GFmLFtC8ub8mv5o39wexij5gjhgBp5/uAwJuuAGuu84rltUkK1b4CPG+ff1T/r594cwzi8+9CcHndaT2GI0bB0uW+P6mTT0Mde7sobJzZ19staTf/Y8/eo/W44/DggVe4vyCC+CUU2CDDar0LVfazJn+mWebNj5ErmHDqr3fPfd478bdd3s4qwohwHvveUgaNQo22gjOOCPzfxZmvorKfvtVr3AcB4WleCgsxRmWvv/eP44cNMg/ShERkXIJwee+DBnicwumTy/7nLp1PTSlC1Lpvm/RAtZbz0PDf/7jFeQGDYJOnar+/cVp0iQvJPDRRz5Ho29fDzGp4WjuXD+2QQNfUyi112j77Ss2HXfFCh/iOGCA36NpU/9f5AUX5OYgjLVr/fPOzz/3XrJs9DKGAD16wOuve5DZa6/MXvuNNzwwf/65D4u8+mofIhl3CfzaTmEpHgpLcYal33/3Acd33RXfAFgRkWpo0iSffD10qH/uVK+ezxHp2ROOPNJ/XrjQH+6Tr6V9n/x50aLS52RccomXg467Kla2FBTAk096meUFC3ybmS+OmgxFXbrAzjtXTW/KmDFeXXDIEJ/bss8+Hpp69MjuhP/S9OnjPYyPPeY9cNmyaJH33K1a5R8YbLRR5a5XUODD+m67zYdObrWVF/s4/fSq7ymTaBSW4qGwFGdYCsH/j3vJJf6RnYiIlOinn7z3aMgQf5gzg7/8xcv69ujhPUCVtXatf46VLlR17Ah77135e1RHc+f6nJxttvEqbM2aZff+CxbAE094cPrxR59HdfbZ3vO15ZbZbUuqzz/3vxM9enhwz/Ywsi+/9F6l/feHt9+uWJny/Hz/b6pPH/8QYrvtPPz17FkzKhTWJApL8VBYijMsgQ9w7trVVwUUkZy0dm3hp+qZVKeOP+BrnH7JZs3yIVlDh/pcEIA99vAHueOOg002ibd9kl0FBT4n7YEHfKgYeKGDCy+Egw7K3JpGUSxeDLvu6m36+uvsV6ZLevhhL2d9yy2+IHJUq1Z5cZI77oCpU7138J//9OCn1Uxyk8JSPBSW4g5LXbr409I778TXBhEpZvp0r/w0bJhP6l+0qGrus9FG6w5n6ty58sNpqrsFC3w40JAh8MEH/jDasaP3IJ1wArRtG3cLJRf8/LMHhUcf9Z6v9u293PTpp2eml7EsJ5/sIf7DDzM7Z6i8QvAiGM8+60HywANLP37FCv+d9e0LM2b43Lvrr/fQmc2wKeWnsBQPhaW4w1L37v6v1VdfxdcGkSxauBDGjvW5CIsX+5CesqpkZcPy5f7QkwxIkyb59k039bkwf/pT5h8kVq0qXH/m228L58q0aVO8vHKultLNlKVL4dVX/eFz2DAvz7vNNt6DdOKJXhlNJJ1Vq3z9mwcegE8+8UqIxx3nQ9M6d/b5VZmuWDhokK8FVN7enKqydKl/2DJ/vj9OpFuraskSrzZ4992+rtY++3jbDz5YvdvVhcJSPBSW4g5LZ58Nb73ly8CL1DDLl/uY+tTqWT/8ULi/fn1/KAZo1WrdgNC5c9Wu1RwCTJhQGI4++sgfuho18hK23br5V4cO2XmQWLKk+O9q2jTfZ+aVxVJ/Nx07elurs5UrfZ7FkCE+pGrFCthiC+896tnTA6oe4qQ8xo/30PTccz73DLyCW/JDmeTX1ltX/O/WDz/4383ddoORI3NnyNq33xZ+uDJyZGFAXLQI+vWD++7zXtuDD/bhdvvtF297pfwUluKhsBR3WOrdG+6801c6U/+3VGNr1hT2kiS/JkzwIVQAm29evLekadN1z/niC/8ffvKcLbZY95xOnSq3xse8eT5MZdgwX97st998+447FoajfffNnUpn8+YV9sIlfz+zZ/u++vV9jkFy6F7yE/RceXAryZo1/iA3ZAi8/LL3Lm60ERx/vPcg7bWX/imUyisogClT1v336KuvPKCDl4bv1GndIbDpemOKWr3ae2R++MGD2RZbVO37KK/Bg31I3tVXw5VXekDq39//O+ve3R85/vznuFspFaWwFA+FpbjD0n/+A5de6gOuW7WKrx0i5VBQAJMnr/sg8vXX3jMDPl+gaC9R1In4y5YV72H58cfC/dtuu+78nl13LTncrFkDo0cX9h6NG+c9Ss2b+6er3brBIYd4kKsOQvBRu6m/m7Fji3+Cvtde/r722Sc3Sv4WFMDHH/sQu+ef9xC4wQbwt795QDrggJq3sKvknjVrYOJE/9Ah9QOdtWt9/6abFv9wpujcp3/8wz/ffPFF//ubi849FwYO9H8XV66EY4/16na77hp3y6SyFJbiobAUd1h67jl/WpgwwT/eFslBM2fCp58WPmCMG+fDxsB7h3bffd2HjLZtMzt8asGCdXtYxowpHLlarx7stFPhvXfayYPbsGHeg7Fkife27LFHYe/R7rvnfg9MVOk+QR871h8MmzTxYpvJ973tttkb1haCt2PoUP9nbuZMf3g78kgfYnfoobkR5KR2W77c/71I/e/n++8L92+9deG/Lc2aeRDp1csLS+SqlSu9p3bDDeHaazXfryZRWIqHwlLcYemDD3yhkPfe849XRXLIrFlw441eOamgwId+dey4bjDaYYd4gsevvxYOTUsGhIULC/dvtVVhz9EBB8RX1jcOS5f6Py3JHrUpU3x7mzb+++jWzStmVcXvZOJEH2I3dKj3CNavD4cd5p8Jde+e/fV5RMpr0SL/QCh1+OuMGb5vhx3835omTeJto9ROCkvxUFiKOyxNmuQTDZ591j9uFckBy5Z5xaS+fX1o3fnn+zj4XXbJ3d6AEPzhfMIE/09qm21UHCBp2rTCeVrvvefzF+rW9bkLyfDUuXPFQ+/UqR6Ohg71OWh16nhA7dkTjjnGhz2KVGezZvnw4I4dYbPN4m6N1FYKS/FQWIo7LC1YAC1bwj33wP/9X3ztEMHH7j/5pJeT/e03X5ywTx8PHlIzrFkDn39eGJ7GjCmcx3XQQYXhqayJ6zNnFi4W+8UXvm3vvb0H6bjjqraSoYhIbaSwFA+FpbjDUgj+Uf3ll/sy2iIxGTYMrrrKewb22APuussffqVmmz/fF91NDtlLzgXbYYfCuU777efDjubN84ntQ4b4mlQheAnlnj19jkSbNvG+FxGRmkxhKR4KS3GHJfCPcA86CJ54It52SK00fryHpHffhXbtPLMfe6yGsNVGIfico+HDPTh9+KFPFm/Y0AtnjB8P+fmw3XaFi8Vut13crRYRqR0UluKhsJQLYalTJ2jd2henFcmSmTN9uN2TT/pE/+uvhwsuyN05SZJ9K1Z4YBo+3Ifa7bWXh6SOHRWmRUSyTWEpHgpLuRCWDj/cZ4+OGxdvO6RWWLLECzfcfbfPUbr4Yl+oUJPwRUREcpfCUjyyv0662eaY9cPsM8yWYxYw26rIMTcltqf7Wlnk2J9KOO7o7L2pSsrLg9mz426F1HD5+fDQQ9C+Pdx2Gxx1FHz3nc9NUlASERERKS6ONdPbA8cD44CPgEPSHPMo8E6RbU0T215Lc/ww4KYi2yZXqpXZlJcHc+b4hAGNbZEMCwHefBOuvtor1e+7L7z+OnTpEnfLRERERHJbHGHpQ0LworJmZ5MuLIUwA5ixzjazU/D2PpXmmvMIYXSmG5o1eXlez3fhQmjRIu7WSA0ybhxceaUvULrttvDyy96jpEwuIiIiUrbsD8MLoaCCZ54GzMZ7kWqW5IIkGoonGfLLL76IbKdOvkhr//7+evTRCkoiIiIiUWU/LFWE2ebAX4BnCCE/zRHdE/OfVmE2ulrNVwKFJcmY336Df/zDe5FeeAGuvRZ++AEuvBDq14+7dSIiIiLVS/UIS3AK3tZ0Q/BeBy4GugEnASuBlzE7OXvNqySFJamgVavgvfd8PlLHjrDppnDnnb5A6OTJcPvtsMEGcbdSREREssGMLcx4wYzfzVhsxktmbBnx3LaJcxeZscyM983olOa4VmY8bsZcM1aY8bkZ3Uq45jlmfGfGKjMmm3FeZd9jtsUxZ6kiTgW+IoRviu0J4eJ1fjZ7GRgN9AEGp72aWS+gFwANGmS0oRWisCQRheAhaNgw//rgA18Lp3592GcfX1C2e3fo0CHuloqIiEg2mdEEGAmswqevBOA24H0zdgmBEtfKMaMl8DGwBDgXWA5cnji3SwhMShzXMHGPVsDVwCzgLOANMw4OgQ9SrnkO8DD+TD4COBB4wAwLgQcz+d6rUu6HJbMuwPbAZZGOD2EtZs8Dd2K2CSH8luaYgcBAAJo2jX+hqRYtoG5dhSVJa+FC7z0aPtwD0i+/+PZtt4WzzoJu3aBrV2jWLNZmioiISLzOAdoB24XADwBmfANMwQPQPaWcez6QB+yfcu5IYCpwM17JGuA4YGfgL8lgZMY7wHigL9Alsa0e8C9gUAj0Tpz7vhmbArea8WgIrMnEm65quR+WPBnnA8+W45zkFPb4g1AUdepA69YKSwL4QrFjxhT2Hn3+ORQUwPrrw4EHwnXXwSGHQNu2cbdUREREcsiRwOhk2AEIgWlmfAIcRelhaQ9gSpFzl5nxEXCEGfVCID9x3ApgVMpxwYzhwBVmbBYCM4E9gY0oPsprEHAGsA/wfiXea9bkdlgyawCcCLxFCHMjnlMPT72/EMKsKmxdZmlh2lpt+vTCcDRiBCxa5FXrOneG3r2996hLFxVpEBERkRLtCLyaZvtE/Nm4NGuB1Wm2rwIaA1vja5iuBdaEUKxDYlXidSdgZqItABPStAWgAwpLpTA7NvHd7onXwzCbC8wlhFEpRx4BtCB9YQcw64kn5beA6Xj34YWJ6/bMfMOrkMJSrTNqFLzyigekSZN822abwd/+5j1HBx0ELVvG20YRERHJFa3qmTE2ZcPAEBLTSlwLYGGaExcAzcu4+GTgYDNahsB8ADPqkBhWl7h28rj1zdghOY8pYc8ixyVfi7ZnQZH9OS+unqXni/z8QOJ1FNA1Zftp+C/1jRKuMw1oDfwb/6UvB8YAhxJC9VqPKS+v8IlZarRFi+Dii2HwYGjUCPbbD84+23uPOnTQOkgiIiKSzrz8EIpXpysi3RSUKE8WDwGXAE+bcQn+TN0bSA76T66T+ixwE/CUGWcBv+FF0/Yrclz1mhJTinjCUgjRHgdDOKqM/aOBAzLQovgle5ZC0NNyDTZyJJx+Ovz6K9x4I1xzDTRuHHerREREpAZYSPoem+ak73H6QwhMNeMkYAD8MW/pS+Be4Eo8FBECi8zogY/6Slap/hEPULcmj2PdHqTUYmstiuzPedVlnaWaLy/PF81ZvDjulkgVWLECLrvMCzQ0bgyffgo33aSgJCIiIhkzkcK5Qqk6AN+WdXIIvAhslji+fQjsDjQDpofALynHfYTPYdoW2CHxugYv/PBlSltI057k4iZltidXKCzlCq21VGONGwe77w733w8XXQRffeXFGkREREQy6DVgDzPaJTeYsRWwd2JfmUJgbQhMCoEfE2W+T4DiayKFQAiBKSHwHdAEL1s+KASWJg75DJgHnFTk1JPxXqVPyvXOYqSwlCsUlmqc/Hy49VbYYw/vMBw+HPr1gyZN4m6ZiIiI1ECPAD8Br5pxlBlH4tXxpuOLwwJgRhsz8s24IWVbfTPuNeNoMw4w42JgLN5DdHfqTczoY8axZnQ142xgHN6zdG3ymMQaStcDp5lxW+LYW4AzgRtCSFt5Lyfldunw2kRhqUb5/ns49VRfI6lnTxgwAJqXVYdGREREpIIS6yIdgM8zGoQXWXgPuCylx4fE9rqs22kSgG2AvwMbAjOAx4Hb0wSbPOA+vMjaHOBl4MYQ1p2HFAIPmRGAK4CrgF+Ai0L4o7BbtaCwlCsUlmqEEOChh+DKK6FhQxg6FE44Ie5WiYiISG2QmFvUo4xjfqJIhbzEgrNHRLzHmeVoz8Ok9GpVRwpLuaJVK6hTR2GpGvv1VzjrLHjnHS8D/thjvm6SiIiIiFRPmrOUK+rW9cCksFQt/fe/sPPOvtDsgAHw9tsKSiIiIiLVncJSLkmutSTVxsKF8Pe/+1C79u3h66/hggu0VJaIiIhITaCwlEsUlqqVESO8N+n55+GWW+CTT2DbbeNulYiIiIhkisJSLlFYqhaWL4dLLoGDD4b11oPPPoPrr4d6mgEoIiIiUqMoLOWSZFgKIe6WSAnGjIHddvP1ki69FL78Ejp1irtVIiIiIlIVFJZySV6ed1ssXVr2sZJVa9bAzTfDnnvCsmU+BO+++6Bx47hbJiIiIiJVRQOHcsnGG/vr7Nk+vktiFQJMnw5ffAF9+3qv0skne6/ShhvG3ToRERERqWoKS7kkdWHa9u3jbUstNHeuB6LUrzlzfF/Lll4e/Ljj4m2jiIiIiGSPwlIuSQ1LUqWWLIFx47zXKBmMfv7Z95nBDjvAYYdB587QpQvssgs0bBhvm0VEREQkuxSWconCUpVYuRLGj1+3x+i77wrraGy1Ffz5z3DRRR6OdttNoyBFRERERGEpt2y0kXdrKCxVWH4+TJrkgSjZa/S//3mBBvA82rkznHiiv3bq5L92EREREZGiFJZySb16PjlGYalC5s3zHqKpU/3n9df3MHT55R6MOneGLbbwPCoiIiIiUhaFpVyjhWkrJAQ47zyvXvfII7DvvrDNNlBHxfFFREREpIIUlnKNwlKFPPMMvPgi9OkDZ58dd2tEREREpCbQ5+65RmGp3KZP9+IMe+0FV10Vd2tEREREpKZQWMo1CkvlUlAAZ5zhhR2efhrq1o27RSIiIiJSU2gYXq7Jy4OlS2H5cmjSJO7W5LwBA+C99+Chh2DrreNujYiIiIjUJOpZyjVaaymy776Dq6/2xWN79Yq7NSIiIiJS0ygs5RqFpUjy8+HUU73z7bHHVA5cRERERDJPw/ByjcJSJLff7gvOPvccbLJJ3K0RERERkZpIPUu5RmGpTGPHwq23Qs+ecPzxcbdGRERERGoqhaVc07q1vyospbVihQ+/a93aizuIiIiIiFSV7Icls80x64fZZ5gtxyxgtlWa40IJX7sWOa4OZtdi9hNmKzEbj1mP7LyZKtCgATRvrrBUgt69YdIkeOIJ/zWJiIiIiFSVOHqW2gPHAwuBj8o49klgzyJf3xc55lbgJqA/cBgwGnges79mrMXZprWW0nr/fbj3XrjgAjjkkLhbIyIiIiJDgOiGAAAgAElEQVQ1nYUQsnxHq0MIBYnvzwYeAdoSwk9FjgvAvwjhn6VcqzUwHbiDEG5M2f4esBEh7FJWc5o2bRqWLVtW7rdRpbp29dVWP/ww7pbkjN9/h112gYYN4auvoGnTuFskIiIikj1mtjyEoCegLMt+z1IyKGVGN6ABMLjI9sHAzpi1zeC9skc9S8VcdhnMmAFPP62gJCIiIiLZkesFHs7HbFVibtNIzPYtsn9HYBXwQ5HtExOvHaq8hVVBYWkdr7wCTz4J114Le+wRd2tEREREpLbI5bA0GLgAOAjoBbQERmLWNeWYFsAiio8lXJCyvzizXpiNxWws+fkZbXRG5OX5uLOVK+NuSezmzIFeveBPf4Ibboi7NSIiIiJSm+RuWArhFEJ4jhA+IoTBwD7Ar8BtKUcZkG7SlZVx7YGE0IkQOlEvB9flTa61NGdOvO2IWQgelBYvhkGDvFCgiIiIiEhpzBhlRk8z6lf2WrkblooKYQnwJtA5ZesCoDlmRcNR85T91Y8WpgXgqafg1VfhX/+CHXeMuzUiIiIiUk3UBZ4BZprR14z2Fb1Q9QlLrmhP0kSgIbB1keOSc5W+zUajMk5hiZ9+gksugf328+IOIiIiIiJRhMA+wM7AUOBsYLIZI8zoYUbd8lyr+oQls/WBw4HPU7a+A6wGTipy9MnABEKYlqXWZVYtD0sFBXD66T4M78knoW65/kqLiIiISG0XAhND4BJgUzwwNQX+C8ww4zYz2kS5TjwTdsyOTXy3e+L1MMzmAnMJYRRmVwLbAe/j85TaAFcCG5MajEKYg9m9wLWYLQG+BE4ADgCOysZbqRK1PCzdfz+MGgWPPQZtq2fxdxERERHJASGwEnjCjHHA/cD+wHXANWa8AFwaAiUWCoirusHzRX5+IPE6CugKTAaOSXxtACwGPgHOIoQvipzbG1gKXIqHqcnA8YTwepW0PBsaNYL116+VYenbb71EePfucMYZcbdGRERERKorMxriHSnnAX/Glxu6As8i3YEb8LlNB5d4jeJVt2uXpk2bhmXLlsXdjOK23dbrZT/3XNwtyZrVq2HPPeGXX2DChMIONhEREZHazsyWhxCaxt2O6sCMHYBzgVOA9YE3gAdC4N0ixx0JPBcCjUu6Vg7WzRagVi5Me9tt8OWX8NJLCkoiIiIiUmETgdn46LWHQ2BGCcdNAcaWdiGFpVy18cbevVJLfP453H47nHoqHHNM3K0RERERkWqsJ/BiCOSXdlAITAL2Le2Y6lMNr7apRT1Ly5d7SNp0Uy/uICIiIiJSCS9QQqeQGY3KUz5cPUu5Ki8PFi70iTwNGsTdmip1zTXw/fcwYgRsuGHcrRERERGRau5RoBHew1TU48AK4KwoF1LPUq5KTtqZU2Ilwxrh3Xehf3+49FI48MC4WyMiIiIiNcCBwKsl7HslsT8ShaVcVQvWWlq40MuDb7899OkTd2tEREREpIZoDcwqYd8cIHIpMYWlXFULwtLFF8OsWfD009C4xIKNIiIiIiLlMhfYqYR9OwMLol5IYSlX1fCw9Npr8MwzcP310Llz3K0RERERkRrkTeB6M3ZM3WhGB6A3vu5SJCrwkKtqeFi64w7Yemu47rq4WyIiIiIiNcz1wMHAV2aMBmYAmwF7AtOBf0a9kHqWclWTJtCsWY0MS+PGwWefwUUXQf36cbdGRERERGqSEJgLdALuAhoDeyRe+wKdEvsjUc9SLquhay317w9Nm3pxBxERERGRTAuBhcB1ia8KU89SLquBYWnuXBgyxBeh3WCDuFsjIiIiIlIy9Szlsrw8X621Bnn0UVi1yofgiYiIiIhUBTN2AM4EtsMXqE0VQqBblOsoLOWyvDz46KO4W5Ex+fnw4IO++GyHDnG3RkRERERqIjM6Ax8CM4G2wESgBbBpYtu0qNfSMLxclpcH8+d7yqgBXn0Vpk/39ZVERERERKpIH+A1YHvAgNNDYHPgUKAuXi0vEoWlXJaXByH4RJ8aoF8/aNMGjjgi7paIiIiISA3WEXgKKEj8XBcgBIYD/wLujHohhaVcVoPWWvrmGxg1Ci64AOrWjbs1IiIiIlKDNQCWhkABsADYOGXfJGDnqBdSWMplNSgs9e8PjRrBWWfF3RIRERERqeF+xBehBfgfcIYZZoYBpwGRH67LX+DBrDXFK0pACL+U+1pSuhoSlhYsgMGD4aSToGXLuFsjIiIiIjXcW8CBwBB8/tIbwCIgH9gQ+L+oF4oWlszWB+4HTgAalnCUBldlWg0JS088AStWqLCDiIiIiFS9EPhnyvfDzdgLOBZoArwTAm9FvVbUnqUBQA/gMbwra1X05kqFNWsGjRtX67C0di0MGAD77gsdO8bdGhERERGpycyoDxwCTAyBnwBCYCwwtiLXixqWugFXEcKAitxEKsjMe5eqcVh66y2YNg3ujFxzRERERESkYkJgjRkv4WXCf6rs9aIWeDBgcmVvJhVQzcNSv36w2WZw9NFxt0REREREaolpwEaZuFDUsDQU6J6JG0o5VeOw9N138O67cP75UL9+3K0RERERkVriLqC3GZUuLRZ1GN5w4D7M1sOrSywodkQIIyvbGEkjLw8+/zzuVlRI//7QoAGcc07cLRERERGRWmRvoCUwzYxPgd+AkLI/hECkBW2ihqVXE69tgdNTb4QP0QuoGl7VyMuDuXO9UkI1Ws118WJ46ik48URo3Tru1oiIiIjUfGZsAdwLHIw/o48ALguBMpf4MaMt8G/gIKA+8AVwVaI4QupxLYEb8FFnmwCzgDeBm0NgbspxT+JrGhV1fwhcVu43Vz4H4fnkd2DHxFeqUOyMEkQNS3+JekHJsLw8KCiA+fOrVep48klYuhQuuijuloiIiIjUfGY0AUbiVatPwwPBbcD7ZuwSAstKObcl8DGwBDgXWA5cnji3SwhMShxnwGvAtnhgmgR0AG4FdjdjrxDWCSJzgSOL3O63yr7XsoTAFpm6VrSwFMKoTN0Qs82Ba4BOQEegMdCWEH5KOaYT0AvYD9gSmAd8BPyTEKYVud5PQJs0dzqGEF7JWLvjkrrWUjUJSwUFPgTvz3+Gzp3jbo2IiIhIrXAO0A7YLgR+ADDjG2AKHoDuKeXc84E8YP+Uc0cCU4GbgeMTx20D7AWcGwIDE9s+MKMAeBAPUalF4VaHwOgMvLfYRO1ZcmYtgD2BFsB8YDQhFJ+/VLr2+C98HB6ADklzzIl4d9l/gInAZsD1wFjMdiWE6UWOHwbcVGRbzajelwxLs2bBzjvH25aI3n0XpkyBwYPjbomIiIhIrXEkMDoZdgBCYJoZnwBHUXpY2gOYUuTcZWZ8BBxhRr0QyAcaJHYvLnL+osRr1OJxVcqMTcs6JgR+jXKt6GHJ7DbgCvyXZImtqzC7ixCuj3wd+JAQ8hLXPJv0YelOQpi7zhazT/AygOfg3X6p5hFCtU6tJUrtWaom+vXzZh93XNwtEREREak1dqSwzkCqiUBZT2VrgdVptq/CR4FtjXdETAQ+BK434wfgO3wY3g3A28nheilamzEP2BDvpXoMuCsE1kZ6RxU3g7LnJUUqBhAtLJldBlyHv8HB+ESujYGTgeswm0sI/4l0rRAKIhwzN822nzGbi/cy1R4bb+yv1SQs/fijL0R7/fVeCU9EREREMqFVPbN1ii0MTBkKBz7ya2GaExcAzcu4+GTgYDNahsB8ADPqAF1Srk0IBDP+CgwCxqSc/ybFA9nX+EiyiUAj4BigDz6U7+wy2lNZvSgelloCh+NTfG6PeqGoPUvnAfcTwv+lbJsMjMJsKXABPmSu6pjtALSGYokVoDtmy/GE+BVwR42YrwSw/vrQsGG1CUsDBnjRvnPPjbslIiIiIjXJvPwQ6FTGQel6UyzNtqIeAi4BnjbjErzAQ2+8EjZAamfHI/iwvfPw5/Id8HlNL5jRPQQ/NgTuK3KPt8xYClxmxp0hMCVCuyokBB4tYVdfM56B6AUgoo4r3ApPjOm8mdhfdczq4X+Ic/HerVSvAxcD3YCTgJXAy5idXMr1emE2FrOx5OdXTZszxazaLEy7dCk8/jgceyxsWuZIURERERHJoIUkeoCKaE76Hqc/hMBU/Dl6d+AH4Fe8TsG9iUN+AzDjcKAncEoIPBwCH4bAw8ApwF/xcuKlGZJ4LSv0VaVBlKNnK2pYmg/sVMK+HRP7q1J/vPLGyYSw7h92CBcTwtOE8BEhvAAcCIzFu/nSC2EgIXQihE7UK1+Ni1hUk7A0eDD8/jtcfHHcLRERERGpdSZSfD0h8DlF35Z1cgi8iE936QC0D4HdgWbA9JR1mpLVxsYUOf2LxOsOZdwm2csVeZ2jKtAKn4cVSdSk8DJwK2bzgaGEsCbR23MccAvwVLmbGZVZH3zc4WmEMLzM40NYi9nzwJ2YbUIIVV7Lvcrl5cGMGXG3olQheLnw3XaDPfeMuzUiIiIitc5rwF1mtEv0FGHGVsDewD+iXCBReCG5ptKmwAn4QrVJsxKvXfAFb5P+nHidWcYt/o4HpaJhK6PM2CvN5gZ4509vfE2pSKKGpWvxNZGeAh7HbAHezVc3cbProt6wXMx643+4lxDCoPKcmXiNM7VmTl4ejBsXdytK9f77MHEiPPGEjxwUERERkax6BLgIeNWMf+LPwbcC04GHkweZ0Qb4EbglBG5JbKsP9AVG4WXBd8Sf/ycCd6fc4yXgX/jcplvxanjbAzcm7vNyyj0GAUPxYX0N8QIPpwMPh8CPGX/36/qY4jkg+YT6Cb6uVCRRF6Vdgtl+eAWJffGgtAD/hb5NCJkPJWaX4KsO9yaEfuU4L9nj9QshzCrr8GohLw/mzPHVXuvkRPn6Yvr3h5Yt4YQT4m6JiIiISO2TWBfpAHye0SA8HLwHXBYCS1MONbzDI/WhMuBV6v6Ol/meATwO3B5CYUnxEFhsxh74+qZXA5vg85leB25Kuc8SPCtcgy92G/Aeq0uABzL3rkt0cJptK4GfQ6Bcw7WiT9jxQPRG4qtyzI5NfLd74vWwRFnwuYQwCrMTgfuAd4CRmO2RcvZiQvg2cZ2e+CJbb+FpNg+4MHHdnpVuZ67Iy4O1a2HBAmjVKu7WFPPzz/Dqq3D11dA48ghQEREREcmkxNyiHmUc8xNFKuQlFpw9IuI9pgNnlXHMAuDoKNerCiHwXqauFVd1g+eL/JxMmKOArsCh+B/ioYmvVMljwBepbY2PpWyBlzkcAxxKCMMy3ejYpC5Mm4Nh6cEH/fX8yB2aIiIiIiJVw4wuwBaJohVF9/UAfgkh2rypksOS2VpgT0L4ArMCSp//EwihPL1Upc9qCeF0fExjWdcZDRwQ+b7VVWpY2jFdkZP4rFgBjzwCRx8NW24Zd2tERERERLgDn7dULCzhFf0uwCtol6m0gHML/DGm7xZqSrGE6ig1LOWYIUN8dKDKhYuIiIhIjugI3FXCvtF4IYxISg5LIdyc8v1NUS8oVSBHw1II0K8f7LQT7L9/3K0REREREQFKX0epDtA06oWilVYzexyztiXsa4PZ41FvKBXQvDnUr59zYemTT+Drr71XSeXCRURERCRHfAd0L2Ffd+D7qBeKWof6dGCjEva1Ak6LekOpADNo3TrnwlK/frDhhnDSSXG3RERERETkDw8D55rRx4x2ZjQwo60ZfYBzgIeiXqg81fBKmrO0MbCiHNeRisjLy6mwNHMmvPgiXHYZNI3ckSkiIiIiUrVC4GEzdsDXgrq6yO7/hMCDUa9VWjW8Y/CVdpNuxmxekaMa44vUjot6Q6mgHAtLDz/sa+RecEHcLRERERERWVcIXGbGA/gCtS2BecC7ITClPNcprWdpSzwIgfcq7QqsKnLMKuBT4Nry3FQqIC8P/ve/uFsBwKpVHpYOPxzatYu7NSIiIiIixYXA95RjflI6pVXDux+4HwCzacAxhPB1ZW4mlZCXB3PmeAm6mKspPP+8N0XlwkVEREQk15hxKtAmBG5Ns+964KcQGBTlWtEKPITQVkEpZnl5sHo1LFoUd0vo1w+22w4OOijuloiIiIiIFHM58HsJ+xYC/xf1QlFLh1+DWb8S9v0Hs6ui3lAqKEfWWvriC/+66CKoE7WWooiIiIhI9rQHJpSwb2JifyRRH3fPAL4pYd/Xif1SlXIkLPXrB+utB6epWLyIiIiI5Ka1+PJG6bQCIs9piRqWtoQSK0dMBdpEvaFUUA6Epdmz4bnn4PTTPTCJiIiIiOSgL4BeJew7FxgT9UJR11laDmxWwr7NKV4lTzItB8LSwIGwZo0PwRMRERERyVG3A++a8QnwKDATzzJnA12AblEvFLVn6SPgKswarrPVf74isV+qUsuWULdubGFpzRp46CHo1g223TaWJoiIiIiIlCkE3gdOALYAHgPeSbxuDhwfAiOjXitqz9JN+HpK32M2mMJ0djK+yNPpUW8oFVSnDmy0UWxh6eWX4ddfvXdJRERERCSXhcCLZrwEdKBwUdpJIRDKc51oYSmE8Zj9BbgLuAbvkSoAPgZ6EML48txUKigvL7aw1K+fL0B76KGx3F5EREREpFwSwWhi6jYztgBOCYHbo1wjas8ShPAFsB9mjYHmwEJCWBG9uVJpMYWlr7+Gjz+Gu+/2kYAiIiIiItWFGU2AY4FTga54NbxIYan8K+WEsIIQflVQikFeHsyaldVbFhTAPfdAkyZw5plZvbWIiIiISIWZcYAZTwKzgCeAnYB7gR2jXiN6z5JZO+B4vIx4oyJ7AyGcFflaUjHJnqUQwCKXhy+3EGDsWBgyBP77X5g50yvgbbhhld1SRERERKTSzNgW70E6BS/osAYYARyGF3f4sDzXixaWzI4Cnsd7ouZQvFR4uSZKSQXl5cGqVbB4MWywQcYvP2ECDB3qXz/+CPXrw2GHwV13wd/+lvHbiYiIiIhkhBnnAafhpcENX0vpTmAInlUWVOS6UXuWbgM+AE4ihLkVuZFkwMYb++vs2RkLSz/+WBiQJkzwonsHHgi9e8PRR0Pz5hm5jYiIiIhIVXoAD0VvAVeEwPfJHWZU+ME5alhqB1yhoBSz1IVpK7HY0cyZPrxuyBAYk1i/eO+9oX9/OPbYwtuIiIiIiFQTHwL7An8FNjPjaeDZEJhTmYtGDUvf4fXJJU6pYamc5s2DF17wHqQPP/R5SbvtBv/+Nxx/PGy5ZYbbKiIiIiKSJSHQ1Yw2+FC8U4B7gL5mDAdeoYLThqKGpauB+zD7nBCmVuRGkgHlDEuLF8Mrr3gP0rvvwtq1sP32cNNNcOKJleqcEhERERHJKSHwM3ALcIsZ++DB6Vi8tykAF5mxOgRGR71m1LB0E96zNAmzKRSfIBUIYf+oN5UKatXKJxWVEpZWrIA33vAepDff9HoQW20FV13lAWmXXaq0kJ6IiIiISOxC4GPgYzMuBo7BK+QdA/Qw47sQopUPjxqW1gKTK9RSyZy6dT0wlRCWPvgAuneHpUu9FsS553pA2mMPBSQRERERqX1CYCVeEW+IGZvgQ/ROiXp+tLAUQteKNE6qQHKtpTRGjPCepREjoGtXz1YiIiIiIgIh8BvQN/EVSfRFaSU3lBKWpk3zQg0HHpjlNomIiIiI1EB1Ih1ltl+ZX1GZbY5ZP8w+w2w5ZgGzrdIc1wizf2P2G2YrEscXv49ZHcyuxewnzFZiNh6zHpHbU92UEpamToW2bbPcHhERERGRGipqz9IHlF1uL+qgr/bA8cA44CPgkBKOeww4HLgKmApcCAzDbE9C+DrluFuBK4HeiWueCDyP2RGE8FbENlUfZfQsde+e5faIiIiIiNRQUcPSX9JsawkcAewPXFSOe35ICF4D2+xs0oUls47A34EzCeGJxLZRwES8HOCRiW2t8aB0ByHclTj7fczaA3fgK/jWLHl5sHy5V3Fo1uyPzcuXe4ZSz5KIiIiISGZELfAwqoQ9L2F2L9AdeDvitQoiHHUksAZ4LuW8fMyGAv/ArCEhrAK6AQ2AwUXOHww8jllbQpgWqV3VRepaSylhaVriXSosiYiIiIhkRiYKPLwJDAUuyMC1knYEphHC8iLbJ+LhqH3i+x2BVcAPaY4D6ADU3LC09dZ/bE6GpXbtYmiTiIiIiEgOMcOA3YEtgUZF94fAs1Guk4mwtB0QpbeoPFoAC9NsX5CyP/m6iBCKzqcqety6zHoBvQBo0KAy7cy+1LCUQj1LIiIiIiJgxvbAy8C2QLrVRgNkMiyZnZpmawNgJ+As4KVI14nOSF9QouibjXrcukIYCAwEoGnTsgpX5JYSwtLUqdCkCWy0UQxtEhERERHJHQ8AjfEaCP/DR6JVSNSepSdL2L4Kn1d0aUUbUIIFeJdZUc1T9idfm2NmRXqXih5XcyTTUJqepXbtwEqPiSIiIiIiNV0n4MwQeKGyF4oaltIN7lpJCOlrWFfeROAYzJoUmbfUAVhN4RyliUBDYGvWnbfUIfH6bRW1Lz7160PLlmnDkobgiYiIiIgwH1iRiQtFW5Q2hJ/TfFVVUAJ4DagPHPfHFrN6wAnA8EQlPIB38PB0UpHzTwYm1LhKeElF1loKQQvSioiIiIgk3A9cYBYx65Si5J4ls1uAhwjh15RtdSKW/i6d2bGJ73ZPvB6G2VxgLiGMIoSvMXsOuA+z+nhFu/PxHq7CYBTCnETp8msxWwJ8iQeqA4CjKt3OXFUkLM2f78suqRKeiIiIiAgbADsAE8wYTvGpOSEEbo1yodKG4fUG3gA8LJnVBVZj1pkQvix3k9f1fJGfH0i8jgK6Jr4/A/gXcBuwITAeODTNvXsDS/F5UxsDk4HjCeH1SrYxd+Xlwdixf/w4daq/qmdJRERERIQbU77fPs3+AJUPS+lKBWSmfEAIZV8nhBXA5Ymv0o5biweq2zLRtGqhSM+SyoaLiIiIiPyhfqYulIl1liTb8vJgyRJYsQIaN1ZYEhERERFJCIG1mbqWwlJ1lLrW0lZbMXWqVxRv1izeZomIiIiI5AozDgX2B1rgFfJGhcCw8lyjrLDUHbOdEt/Xwcf3HYnZrsWODOHx8txYKqFIWFLZcBERERERZ0ZT4HU8KAVgIb4O6zVmfAB0D4HlJV+hUFlhqXeabTek2RYAhaVsSQ1L+Jylzp1jbI+IiIiISO7oA3QBzgSGhMBqM+oDfwf6A7cDl0W5UGlhSX0VuSolLK1dCz//DMcfH2+TRERERERyRA/gnyHwVHJDCKwBnjKjBXAFlQ5LIfxcyUZKVWnd2l9nzWLGDMjP1zA8EREREZGElsCEEvZNAFpFvVClV7WVGDRsCBtuCLNn/1EJTwvSioiIiIgA8DNweAn7DgV+inohVcOrrhJrLWlBWhERERGRdQwE/m1GE+AZ4DdgY+BE4Fzg6qgXUliqrjbe+I+epbp1YYst4m6QiIiIiEj8QuBuM/KAS4GzU3blA3eHwD1Rr6WwVF3l5cHXXzNtmgel+hlbp1hEREREpHoLgavN6Avsia+ztAD4LATmlec6CkvVVcowPA3BExERERFZVyIYvV6Za1Q8LJl1AHYAPiOEXyvTCKmAvDz4/XemTQscfrjF3RoRERERkdiYsRcwPgSWJb4vVQh8GuW60cKSWX+gHiGcl/j5b8BzQF1gMWYHE8KYSNeSzMjLYzmNmTXL1LMkIiIiIrXdx8AewBeJ70MJx1liX90oF43as3QYcHPKzzcDbwA3AHcDNwJHRLyWZEJeHj+xFaBheCIiIiJS6x0MfJv4/hBKDkvlEjUsbUyyHrnZ5sCOwFmE8D/M/gM8lonGSDnk5TENT0laY0lEREREarMQeC/l+xGZum7URWlXAM0S3+8PLAbGJn5eCqyXqQZJRClhST1LIiIiIiLOjO/N2KWEfTua8X3Ua0XtWfoSuBCzX4ALgXcJoSCxry2+0JNkU14eU2lHk/qrad26QdytERERERHJFe2BRiXsawxsHfVCUcNSb+AdYDywCDgvZd/R+EQqyaZGjZhWbxvarjcfs03ibo2IiIiISC4pac7Sn/A8E0m0YXhe6W5LoAvQlhC+Sdk7EC/wIFk2tc42tG2kqu0iIiIiAmZsYcYLZvxuxmIzXjJjy4jntk2cu8iMZWa8b0anNMe1NON+M6aascKMaWb0N2OjNMcebcZXZqw042cz/mkWrQpdeZlxaaJNU/Gg9Ery55Sv34CHgOFRrxt9naUQlgHjirSqJSG8GfkakjEhwLS1W7C/DQN2j7s5IiIiIhIjM5oAI4FVwGl4YLgNeN+MXUJgWSnntsTLbS8BzgWWA5cnzu0SApMSxxnwGrAtXhV7EtABuBXY3Yy9QvAeHTO6AS/iheAux3t0bsdrHVyT2XcPwC/AJ4nvtwK+AeYVOWYVXjFvYNSLRl1n6RxgQ0L4d+LnnYG3gU0w+wo4ghBmRb2pVN6CBbBkbVPa5U+OuykiIiIiEr9zgHbAdiHwA4AZ3wBT8AB0Tynnng/kAfunnDsSmIovGXR84rhtgL2Ac0P4I3B8YEYB8CAeopIPp3cAH4dAr8TP75vRDPinGfeGQEazQwi8DLycaDvADSEwrbLXjVoN72K8Il7SPfhYv8uADYBbKtsQKZ+pU/217bKJ8TZERERERHLBkcDoZNgBSISFT4Cjyjh3D2BKkXOXAR8BR5j90cGSrCq2uMj5yTlAdcCHAwK7AoOLHDcIqI+v4VplQuCUTAQliD4Mb0vgOwDMNsDLhx9NCG9hNh/ok4nGSHTTEn/8bZd+A2vWQP368TZIREREROK0I/Bqmu0TgePKOHctsDrN9lUUVo+bnLjWh8D1ZvyA54MO+JC8t5PD9RJtAZiQerEQmGbG8sQ5VSoR8LoB21G8Ml4IIVp+iRqW6gLJUuH74GMgP0j8PB1oHfE6kiF/hCWmwZw5sNlm8TZIRERERKpQq3pmf6xzCjAwZSgcQAtgYZoTFwDNy7j4ZOBgM1qGwHwAM+rgxd2S1yYEghl/xXuIxqSc/ybrBrIWidd07VmYsknoPcMAACAASURBVL9KmLEJHuq2xnOLJXalVsiLFJaiDsObAhye+P5E4FNCWJ74eVP8D0GyaOpUaLX+KtZjKcyeHXdzRERERKRKzcsPgU4pX+mKFKQrl21pthX1EJ4LnjZj60TY+A++nioUdpoAPIIP2zsPH212HtAJeCERsFLvWdH2VFZffGhgu8T99sLnU90J/IDPvYokas/SXcAg7P/bu/Mwuapq7+PflXTGzpxABzISgUCQAMoQRQQDCjIrQV6GC4gXEBFfuC+iCAoBhasYuFz0IoMgYgQZJcAFAQlhDIPIFDAMXUnIBJkDGTrp9Hr/2KfSlepT3dVJd51TXb/P85znVJ2pVvdJpWvV3nttO4WQmeZmjl8hVJuQEspkYLsh60OPUSVLIiIiIpWuUItNf+JbeDZyp9aME4HfwsZxS68C1wDnAwsAzDgMOB44yJ2/R8c9HZXrfgw4gtAVMNuQEhdPP9q/oeXLwAWEHnAA66PxWD+JKvpNAr5RzIWKnWfpz4TM8UrgK7jfl7P3I+C64uKWtpLJwKhR0RMlSyIiIiKVbgaNY4VyjSGUy26WO/cCQ6Ljt3fn80Av4EN35kSH7RqtX847/aVovXNOLOTHY8ZIoGcx8WyhQcA8dzYAqwgJWtYTwPhiL1RsNzxwfxb3Sbg/nbf9Etz/t+jrFMvsKcy8wPJodMzIZo7p1/wLlK8NG2D2bNhup6ggiZIlERERkUo3BRhnRvbr9Gxysm+0r0XubHDnHXc+MGNb4DhCSfCsbLnvvfNO3Sdaz4uuMwd4HTgx77iTgPWEKYja0zxgYPS4Fvhqzr49gbXFXqj4SWnNegKnEVqYBgBLCEUe/pAzfqktfQ/ok7ftC4Sy5fk3/MqYbZ+0Q0ypMG9eKIC33Y5dobpayZKIiIiI3AR8H3jAjIsJ44UuJ3RFuyF7kBkjgA+Ay9zD9D9mdCGM85lGGOSxC3AhoYVoUs5r3Af8gjC26XJCNbydgEui17k/59ifAA+ZcQNwB2FS2ouBa9t6jqUYUwk5ywOECWivM2MsIVE7FLi52AsVOyntYEJitCMwm5BVjgImAOdgdgDubfuJ3b1p81yYHHcdcGfenlrcp7fp66dYthLeqFFATY2SJREREZEK584qM8YTxhndTihs8HfgXHc+zTnUCJWuc3uYOaHowQmELmtzgVuAK9wbS4q7s9KMccClhDFB2xDGMz0IXJr7Ou78rxkTCInUqYShO1cQkq329lOiliV3fmtGV0IrWU/C7+fSYi9UbMvSrwiDw/bD/bmNW82+CNxLqCxxarEvulnMehAKSzyIe0VX39s4Ie12KFkSEREREWBj97djWjhmFnkV6dypBw4v8jU+BL5T5LH3EVqjSsqdj4GPc55fQ0iSWq3YMUtfBy7cJFEKr/w8oTntsLiT2tg3gd7AbTH7rsSsHrMVmE3BbNeYYzqMTAY6dYLhw1GyJCIiIiLSToptWeoFzC+wb260v72dTMgQcweE1RH6YD4GLCL0mfwJ8Dxme+P+TpOrAJidAZwBQNeu7RdxO8lkYNgw6NKFkCw991yL54iIiIiIdFRmsfNOFeLunFnMgcUmSzOBfwMejdl3EmFwV/sx2xY4CLgW9/qN290XECbCynomqpQ3A7goiq0p9xsh+oVWV8dNlpVqtbVRFzwIydLixVBfD1XF1+sQEREREelADmXTSXB7E4rFNRDmmepP6FW3MlqKUmw3vF8Dx2P2BGanYfZ1zL6N2d8IA8GuKvYFN9NJhFjjuuBtyv1D4Flgr3aOKTGbzLFUUwPusGhRojGJiIiIiCTFnaHuDHNnGPAtQkJ0EtDDna2AHoTGn5XR/qIU1xTh/qeodPhlbFpq7yPgu9Gkte3pZOB13F8v8nhj08yyw1izBhYsyGtZgjBuaZttEotLRERERCQlrgF+5c7GHMWd9cBkMwYA19I4N1SzWjMp7Y3AtoS66/tF6yG431R83JvBbM/otVpuVQrHDydMvvViO0aVmFmzwjo2WRIRERERkd0Iw4jizASKLgbXukEu7g3ApkUTzA4CrsZ9bKuuVbyTgXqgaeuV2SRCwvcCocDDaMIEWg2EOu4dziZzLIGSJRERERGRTX1EmA/28Zh9x5JTVrwlbVERoC+h5aftmXUBjgceLTDp7QzgLMIcT72BxcCTwETcC2WTZW2TOZYABg8OayVLIiIiIiIQutlNMmMwcDcheaohjFU6DPh/xV4o3eXT3NcDWzWz/xbC7MIVI5OBHj0aG5To1StsULIkIiIiIoI715ixGvgpcETOrvnAWe7FlxlPd7IkTWQyoVXJsvMum2liWhERERGRHO7cYMZNwAhgG2ABMNudhtZcR8lSmdlkjqUsJUsiIiIiIpuIEqNMtGyWwsmS2aiC+zY1eHNfXFrHPbQs7bdf3o6amsYyeSIiIiIiFcaME4BH3VkaPW5Wblnx5jTXsvQ+xc1V1GHnNEqbZctg5cqcSnhZNTXwYoeslC4iIiIiUow/AeOAl6LHzXHiKm3HaC5Z+nZxcUmpNKmEl1VTA4sWwYYN0LlzyeMSEREREUnYDsCHOY/bROFkyb24SWClZLJzLMUmSw0NsGQJbL11yeMSEREREUmSOx/EPd5SndrqQtL+mk2WQEUeRERERETakKrhlZHaWhg4EPr0yduRmyztumvJ4xIRERERSZIZ71F8HQV3Z3QxBypZKiPZOZaaUMuSiIiIiFS2F2mHonNKlspIJgN77BGzQ8mSiIiIiFQwd05qj+tqzFKZ2LAhTKUU27LUty907apkSURERESkDallqUzMnw/r1xdIlsxC65KSJRERERERAMzYBRgNdM/f1xaT0kqKZOdYajIhbZaSJRERERERzOgLPAjsm90UrXPHNBWVLKkbXpkoWDY8S8mSiIiIiAjAL4DBwHhConQs8DXgL0AtMK7YCylZKhOZDHTqBMOHFzhAyZKIiIiICMAhwBXAs9HzWe484c4JwFTg7GIvpGSpTNTWwtChoY5DrJoa+PhjaGgoaVwiIiIiIimzLfC+OxuAtUDvnH13A0cUeyElS2Wi4BxLWTU1UF8Py5aVLCYRERERkRT6COgXPZ4N7JOz7zM0jmFqkQo8lIlMBg4+uJkDcudaGjiwJDGJiIiIiKTQs4QE6SFgMjDRjOFAPXAa8HCxF1KyVAbWrAmlw1tsWYKQLI0ZU5K4RERERERS6DJgSPT4V8BWwHFAD+AR4PvFXkjJUhmYPTusi06WREREREQqlDvvAe9Fj9cB/zdaWk1jlspAtmx4wTmWQMmSiIiIiFQsM24x48ttfV0lS2UgOyFtsy1L/ftDVZWSJRERERGpRMcBU83ImDHRjO3b4qJKlspAJgPdu8Pgwc0c1KkTbL21kiURERERqUQ1wL8Ds4CLgZlmPGvG6Wb03dyLKlkqA9my4dZSkcOaGli4sCQxiYiIiIikhTufunOrO18BRgI/BQYANwALzLjDjK+btS7/UbJUBmprW+iCl1VTo5YlEREREalo7nzozhXujAHGAbcABxJKic8z49fFXkvJUhlocULaLCVLIiIiIiIbufOSO98nlBK/BtgaOK/Y81U6POWWLYMVK1qohJdVUwMffwzuRfTZExERERHp2KJCDycDJwEjgE+Au4s9P70tS2YHYOYxy/K84/pjdjNmizFbhdkTmO2aUNRtrqhKeFlDhsC6dfDii+0ak4iIiIhIWpkxwIyzzHgBmAn8BHiXkDANduf0Yq+V3mSp0Q+AL+QsB23cY2bAFOAQ4BzgGKALMBWzoSWPtB1k51gqKlk68UQYMQKOPTa0MImIiIiIVAAzupjxDTPuB+YDvwV6AxcCw905xJ073FnbmuuWQze8d3CfXmDfkcCXgPG4TwXA7AUgA1xASLTKWqtalgYOhPvug333heOOg8cfD3MviYiIiIh0bAuBfsBS4CbgNnde2dKLlkPLUnOOBOZvTJQA3FcADwJHJRVUW8pkYMAA6FtsdfjPfQ5+9zt46in48Y/bMzQRERERkbR4BpgAbOvOOW2RKEF5JEuTMduA2RLM/ozZ8Jx9uwBvxZwzAxiOWa/ShNh+iq6El+uUU+Dss2HSJPjLX9olLhERERGRtHDnaHfud2d9W143zcnSCmASYSbe8cDlhPFKL2C2dXTMAGBZzLlLo3X/9g6yvdXWFlkJL9/VV4fueKedBm++2eZxiYiIiIh0dOlNltz/ifv5uD+I+zTc/4tQyKGGxrFIBnjM2c3XzTY7A7NXMHuF+vo2DbstNTTA7Nmb0bIE0LUr3H039OkD3/gGLF/e8jkiIiIiIrJRepOlOO6vEsr+7RVtWUpoXcqXbVGKa3UC9xtx3xP3PdNcAGH+/FAJfLOSJYBttoF77gkZ10knhexLRERERESKUl7JUpDbmjSDMG4p3xhgDu6fliyqdpCthLdZ3fCy9t0Xrr0WHn4YLr+8TeISEREREakE5ZUsme0J7AhkZ12dAgzBbP+cY/oAR0T7ylqr5lhqzllnhaIPl14KDz20pWGJiIiIiFSE9PZBM5tMmC/pVWA5sAdhUql5wHXRUVOAF4A/YfZDQre7CwmtT78qdchtLZMBszDP7BYxg+uvhzfeCN3xXn4ZdtihTWIUEREREemo0tyy9BZhHqVbgb8B5wL3AfvgvhgA9wbgcOBx4H+A+4ENwFdw/zCBmNtUbS0MHRpqNWyxHj3ChLVVVfDNb8KnZd1DUURERESk3Zl7XDG5ylFdXe2rVq1KOoxY++0HnTrBtGlteNHHH4dDDoEJE+DOO0Ork4iIiIikmpmtdvfqpOOoNGluWap4mcwWFneI89WvwhVXwF13hbmYREREREQklpKllFq7FubNa4PiDnEuuACOOSasn3yyHV5ARERERKT8KVlKqdmzw7pdkiUzuPVWGD0ajjsO5sxphxcRERERESlvSpZSKls2vM274WX17g333w91daGVae3adnohEREREZHypGQppbIT0rZLy1LW6NHwxz/CK6/A2WdDhRf7EBERERHJpWQppTIZ6NYNBg9u5xc6+mi46CK45Ra46aZ2fjERERERkfKhZCmlamtDq1KnUtyhiRNDOfHvfx+mTy/BC4qIiIiIpJ+SpZTKZNq5C16uzp1h8uQwA+4xx8BHH5XohUVERERE0kvJUkqVNFkCGDAgFHxYtgy+9S1Yv76ELy4iIiIikj5KllJo2TJYvrwdK+EVsttuYdzS00/DD39Y4hcXEREREUmXqqQDkKayZcNL2rKUdeKJ8PLLcO21sNde4bmIiIiISAVSy1IKJZosAVx1FXz5y3D66fD66wkFISIiIiKSLCVLKZSdY6nk3fCyunSBu+6C/v3hG9+ApUsTCkREREREJDlKllIokwl5St++CQZRUwP33gtz58IJJ8CGDQkGIyIiIiItMWOYGfeYscKMlWbcZ8bwIs/dLjp3uRmrzJhqxp55x5xqhjezDM459qkCx5zb1j93e9KYpRQqeSW8QsaNg+uug+9+FyZMgKuvTklgIiIiIpLLjJ7Ak0AdcArgwM+BqWaMdWdVM+cOBJ4FPgHOBFYD/xGdu7c770SHPgx8If904EGg1p2FefveiK6Xa1Yrf7REKVlKodpaGDs26SgiZ5wRSvNddhnstBP84Adw0UXQr1/SkYmIiIhIo9OBUcBod94HMOMN4D1CwnJ1M+eeBdQA++ec+yRQC0wEvgXgziJgUe6JZuwHDAQuibnuJ+5M34KfKXHqhpcyDQ0wa1aKGnDM4Ec/gnffDd3xJk2C7beH3/xGczGJiIiIpMeRwPRssgPgTgZ4DjiqhXPHAe/lnbsKeAY43KzZBpZTgHXAnZsbeJopWUqZBQtg3boUJUtZQ4bArbfCq6+G+ZjOOQd23RWmTAH3pKMTERERqXS7AG/FbJ8BjGnh3A2EhCdfHdAD+EzcSWb0AI4FHnJnScwhe0Tjp9ab8YYZ32khjtRRspQyiVfCa8nuu8MTT8CDD4ZWp6OOgvHjQxIlIiIiIu1kUJUZr+QsZ+QdMABYFnPiUqB/CxefCewQjV0CwIxOwN45145zNNAHuC1m39PAuYQWrwmE7oA3m3FxC7GkipKllEl8jqVimMHhh8Mbb4TueG+9BXvuCaeeCvPmJR2diIiISAe0uN6dPXOWG2MOiuvuY0Vc/HeEvOCPZnzGjG2A/wayn0gbCpx3CmEM0/82CcT5mTs3uTPNnQfcOQb4K3CRGb2KiCkVlCylTCYTcpERI5KOpAhdusDZZ8P778MPfwh33AE77AA/+xl8+mnS0YmIiIhUkmXEtwD1J77FaSN3aoETgc8D7wPzCVXvrokOWZB/TpRQHQRMdqe+yBjvALoDuxZ5fOKULKVMbW0YHtStW9KRtELfvvDLX8LMmaFb3uWXh6Tp5ps1P5OIiIhIacwgjFvKNwZ4u6WT3bkXGBIdv707nwd6AR+6MyfmlJOAzsR3wSsk28pVNgPelSylTGrmWNocI0eG1qUXXgg/xOmnwx57wOOPJx2ZiIiISEc3BRhnxsaR72aMBPaN9rXInQ3uvOPOB2ZsCxwHXF/g8JOBN9x5rRUxngCsAd5sxTmJUrKUMrW1KS7uUKxx4+C55+Cuu0J3vK99DQ49FGbMSDoyERERkY7qJsKErw+YcZQZRwIPAB8CN2QPMmOEGfVm/CxnWxczrjHjaDPGm3EO8AqhtWpS/guZ8TngsxRoVTJjPzMeNuM7ZhxoxjfNeIBQ7GFicxPkpo2SpRSpq4P588u4ZSmXGRx7LLzzDvz61/D882Gm3e9+Fz76KOnoRERERDqUKAEZD7wL3A5MBjLAeHdyB5Mboftcbh7gwA6EpOoRQhW7W4CD3WNLip8C1EevEWdBdP3LCMUf/ghsBZzgzi835+dLinmFz5FTXV3tq1alI7l9910YPRpuuw1OPjnpaNrY4sVw2WVw/fXQowf8+Mdw3nnhsYiIiIg0y8xWu3t10nFUGrUspUjq51jaEoMGwX//d+iKN348XHQRbL89TJyocuMiIiIikkpKllKkLOZY2lI77gh//StMnQq77hqSpeHD4eij4ZFHVD1PRERERFIjvcmS2QTM7sVsNmZrMJuJ2ZWY9c45ZiRmXmDpl1zwmyeTCSXDt9km6UhK4IAD4NFHwxxNF1wQKugdeih85jPwi1/Agibl/EVERERESiq9Y5bMpgNzCFU85gJ7AJcC/wK+iHsDZiMJA9eupGlJxJdxb7GZIk1jliZMgLfegn/9K+lIErBuHTzwAPzud/Dkk1BVBUceGQpCHHggdEpvXi8iIiLS3jRmKRlpTpa2wn1R3raTCSUKD8T9yZxk6XTcb96cl0lTsvT5z8PWW4feaBXtvffgxhvh1lthyZIwiOuMM+Db3w6/IBEREZEKo2QpGen9uj4/UQpejtZDShlKqZT1hLRtaYcd4KqrQuGHP/8Zhg4N1fOGDoXjjgstT2lN8kVERESkw0hvshRv/2j9Tt72KzGrx2wFZlMw27XUgW2p5cth2bIOWglvc3XrBscfD9Omwdtvw9lnw+OPh255o0fDpEmhJLmIiIiISDson2TJbAhhYqsncH8l2lpHmDzrTOArwPnArsDzmO3czLXOwOwVzF6hvr594y5SRVTC2xI77wzXXBNam/74R9hqKzj/fBgyBE48EZ5+Wq1NIiIiItKm0jtmKZdZL+ApYFtgb9znNnPsMGAGMAX3k1q6dFrGLN13HxxzDPzjH/C5zyUdTZl46y244Qa4/XZYsSIkVGecAUcdBSNHglnSEYqIiIi0CY1ZSkb6W5bMuhMq3Y0CDm42UQJw/xB4Ftir/YNrOx16Qtr28tnPwnXXhdam3/8eeveG884Lv8TBg0PSdMUVYYzTJ58kHa2IiIiIlJmqpANollkX4F5gb+Ag3N8s9kygDJrMGmUy0K9fWKSVqqvhtNPCMmNG6JL34oswfTpMiSrKm4Xkap99YNy4sOy8s0qSi4iIiEhB6e2GZ9YJuBM4EjgM978Xed5w4C3gftxPaenwtHTDO/RQWLgQXn016Ug6mKVL4aWXGpOnF18MlTQgtETtvXdj8rTPPmEslIiIiEjKqBteMtKcLF0PfBf4BfBQ3t65uM/FbBKhK+ELwCJgNHAh0BfYB/eZLb1MWpKlnXYKDR/33JN0JB2ce5jHafr0xuWNN2BDNH/xqFGNidO4cbD77tC1a7Ixi4iISMVTspSMNCdLs4ARBfZOxP1SzE4DzgK2B3oDi4Eno/0tJkqQjmSpoQF69oRzzgnTC0mJrV4dKmtkW55eeAHmzw/7unWDPfaAL3wBDjgA9tsP+vdPNFwRERGpPEqWkpHeZKlE0pAszZsX5lv97W/he99LNBTJmju3MXmaPh1efhnq6sLYp913D4mTkicREREpESVLyVCylIJk6dlnw2fuRx6BQw5JNBQpZO3aMPbpqafC8vzzSp5ERESkZJQsJUPJUgqSpdtvh5NPhn/9C0aPTjQUKZaSJxERESkhJUvJULKUgmRp4sSwrF4N3bsnGopsLiVPIiIi0o6ULCVDyVIKkqVTT4UnngjDZKSDaC552m23xuRp773DBLpmycYrIiIiqaZkKRlKllKQLO2/f6iI98wziYYh7alQ8gSw9dah9Sl32XFH6Nw5yYhFREQkRZQsJUPJUgqSpWHDYPx4uO22RMOQUlq7NlTY++c/4bXXwvLWW7B+fdjfo0eYeCs3gRo7Fnr1SjZuERERSYSSpWRUJR1ApaurC6XDt9su6UikpLp3D+OX9tuvcdu6daHKRzZ5ev31MEvxTTeF/Waw/faNydNuu4X1ttuqG5+IiIhIO1CylLA5c8BdyZIAXbuG1qOxY0N5RAj/OObObUygXnsNXn0V7r678bxBgzZtgRozJkzcNWiQkigRERGRLaBkKWG1tWE9alSycUhKmYV+msOGwRFHNG5fuRLeeGPTJOq66xrHQUFIvrbdFoYM2XTJ36YSjCIiIiKxlCwlLJMJa7UsSav06QNf+lJYsurrYebM0JVv3rzGZf78kEw9/DDEjc8bMKDlhGrQIOjUqXQ/n4iIiEgKKFlKWCbT2AAgskWqqmCXXcISxz20SOUmUtlkKvv49ddh4cJwbK6uXUPXvuHDwzJixKbrYcOgZ8/2/xlFRERESkjJUsJqa2HkSH1pLyVgBn37hmXMmMLH1deHhCk3mZo7Fz78EGbPhqlTw/aGhk3P22qrwsnU8OFhv8ZQiYiISBlRspSwTEZd8CRlqqpCK9LQoYWPqa8PCdOcOSGBmjOn8fHMmfDYY027/HXv3jSZGjo0JFG5S+/eSqpEREQkFZQsJSyTgb32SjoKkVaqqgoJz4gRm5Y/z3KHZcvik6k5c8L4qYUL46/dtWvTBKq5pX9/JVciIiLSLpQsJWjFCli6VJXwpAMyC4UjBgwI5czj1NXBggWwaFHzywcfhPUnn8Rfp6oqFKDITaC23hpqajZdZx9Xaz4/ERERKY6SpQSpEp5UtG7dwoC9kSOLO37tWli8uOXk6tVX4eOPw7cRcXr2LJxI5a8HDNCAQhERkQqmZClBSpZEWqF795bHUuWqqwvJ00cfheQpf/3xx6Fb4Msvh8cbNjS9RufOm7ZU1dTA4MGbrrOPBw4Mx4uIiEiHoWQpQZqQVqQddetWfHLV0BDGWMUlVNnHCxfCe++F9dq1Ta/RqVNjUpWfUOWv1WIlIiJSFpQsJSiTCVWc+/dPOhKRCtepU2gZGjiw+bLqEIpXfPJJSKIWLiy8fuedsF63ruk1qqoaE6tBgxpfe+DAkEjlPs8uffuqkIWIiEiJKVlKUG2tuuCJlB0z6NMnLDvs0Pyx7mHsVHNJ1ZIl4ZuTJUtg+fKmEwJnde4cvlmJS6TyE60BA6Bfv7D07q1WLBERkc2kZClBmUzLX2KLSBkza0xadtqp5eM3bAjdAZcuDclT/pK7fc4c+Oc/w+M1a5qPoU+fxjj69QutVHGPCz2v0p8KERGpTPoLmJCGBpg1Cw47LOlIRCQ1OncO3fIGDWrdeWvWbJpILV0aWrSWL29c5z6ePRtefz08X7GicGtWVnV1SJp69QrVBKurwxL3uNht2cddumz+70tERKSdKVlKSHaMuLrhicgW69EDhgwJS2s1NIQxWIUSq9znq1aFZfVqWLkyzJO1enXj9lWrwvVao1u30L0wtzWr2Od9+yrZEhGRdqVkKSHZsuGqhCciierUKSQdfftu+bXcQ0GL3AQq+zhu26pVjYna8uWhC+KSJWEi4uzz+vrmX7O6umkyle06aBZ+vvx13LZi1t27N45X69u38XHu0r27CnGIiHQgSpYSojmWRKTDMQstRdnWoi3lHhKr3BauZcuafz5vHrz9dmjhamgI18h/3Ny25vYVo6qq5YQqd+ndO3S/zE/icpf8bS0979QpxNGtG3Tt2nhPunULLXFK5kREiqZkKSHZOZZGjkw0DBGR9DJrHN+0OV0M21pdXeh+2NyyYkXTbQsXwrvvNj5vriBHKeQnUPnPC23r1q3xfvTq1fRx3Lbq6nAtEZEypWQpIZkMbLtt6LEhIiJloFs32GqrsGyJ9etD98OVK8M625KVv+S2cjW3LX/7hg2h++K6dSHByy6teb5mTUj8cvevXRta+j79tOWiILm6dGk+yerRIyRU2SWbrG3Jti5dQotdVVVYF1qy3TVFRAroGMmS2TDgGuCrgAFPAOfiPifRuJqRyagLnohIRerSJcyFNWBA0pFsHveQOH36aePYs+zj1mxbtCj8McxN1rJLXV3rErItYdZ8MtVcsrWlxzd3zpZur6oqvHTp0vz+QkvnzuG+5C/ZfxeFlpb2t2bJfjnQ0gJNu6du6WKm5LoClX+yZNYTeBKoA04BHPg5MBWzsbivSjK8Qmpr4YADko5CRESklcxCa1CPHlveytacbOtY/pKfWBXalm1h27Ch8NLS/mKPyT+2rq715zS3T9Lj0kvhHzddGAAADs5JREFUkkuSjkJKqPyTJTgdGAWMxv19AMzeAN4DzgSuTi60ws45p7g5KkVERCpStjWjZ8+kI0letntlSwlW/vb6+k2X9eubbmvtkm1dyV+g8L5i9hez5LbutLRA4W6rm7ts2AD775/MvwFJjHmpmrnbi9nfge6475u3fRoA7s3+q66urvZVq1LZ+CQiIiIiAoCZrXb36qTjqDSdkg6gDewCvBWzfQYwpsSxiIiIiIhIB9ERuuENAJbFbF8KxE/0YXYGcAagkqYiIiIiIhKrI7QsQSjqkK9wuRL3G3HfE/c9qeoI+aKIiIiIiLS1jpAsLSO0LuXrT3yLk4iIiIiISIs6QrI0gzBuKd8Y4O0SxyIiIiIiIh1ER0iWpgDjMBu1cYvZSGDfaJ+IiIiIiEirdYTS4dXA68Aa4GLC+KXLgd7AWNw/be50lQ4XERERkbRT6fBklH/LkvsqYDzwLnA7MBnIAONbSpREREREREQKKf+WpS2kliURERERSTu1LCWj/FuWRERERERE2oGSJRERERERkRhKlkRERERERGIoWRIREREREYmhZElERERERCRGxVfDM7MGwhxNzakC6ksQjrQN3a/yoXtVXnS/yofuVfnQvSovSd6vHu6uho4Sq/hkqRhm9oq775l0HFIc3a/yoXtVXnS/yofuVfnQvSovul+VR9mpiIiIiIhIDCVLIiIiIiIiMZQsFefGpAOQVtH9Kh+6V+VF96t86F6VD92r8qL7VWE0ZklERERERCSGWpZERERERERiKFkSERERERGJoWSpADMbZmb3mNkKM1tpZveZ2fCk45KmzOwAM/OYZXnSsVU6MxtqZteZ2Qtmtjq6LyNjjutuZleZ2QIzWxMd/+XSR1y5WnGv4t5rbma7lz7qymRmE8zsXjObHb1fZprZlWbWO++4/mZ2s5ktNrNVZvaEme2aVNyVqJh7ZWYjm3lf9Usy/kpjZgeb2ZNmttDM6sxsrpndZWZj8o7TZ8QKUpV0AGlkZj2BJ4E64BTAgZ8DU81srLuvSjI+KegHwMs5zzXJX/K2B74F/AN4BvhageN+DxwG/BCoBc4G/mZmX3D310oRqBR9rwD+ANyQt+3d9glLYpwPzAF+AswF9gAuBb5iZl909wYzM2AKsB1wDrAMuJDwd2x3d5+bSOSVp8V7lXPslYR7luuTUgQpGw0g/B/4P8AiYDjwY2C6me3q7rP1GbHyKFmKdzowChjt7u8DmNkbwHvAmcDVCcYmhb3j7tOTDkI28bS71wCY2b8T8wHczHYDTgBOc/dbo23TgBnAZcCRpQu3orV4r3LM03stUUe4+6Kc59PMbClwG3AA4YPckcCXgPHuPhXAzF4AMsAFhC+XpP0Vc6+yavW+Spa73wHckbvNzF4C/gVMACahz4gVR93w4h0JTM++CQDcPQM8BxyVWFQiZSbvW9NCjgTWA3/JOa8euBM42My6tVN4kqPIeyUpkPfhOyvbqj4kWh8JzM8mStF5K4AH0d+xkinyXkm6LYnW66O1PiNWGCVL8XYB3orZPgMYE7Nd0mGymW0wsyVm9mf1Hy4buwAZd1+dt30G0JXQPUzS5ayoP//qqH//fkkHJOwfrd+J1s39HRtuZr1KEpXEyb9XWVeaWX00DmaKxpclx8w6m1lXM9uB0OV4IeELPNBnxIqjbnjxBhD6d+dbCvQvcSzSshWEpvFpwEpCn/CfAC+Y2R7u/nGSwUmLmnu/ZfdLevwJeAiYD4wgjDN70sy+6u5PJRlYpTKzIYQuq0+4+yvR5gHArJjDs++r/sCn7R+d5Cpwr+oIH8gfI4yT2YnwN+x5M9vb3fOTKml/LwKfjx6/T+jOmv0soc+IFUbJUmFxs/VayaOQFrn7P4F/5myaZmZPAy8R+uVfnEhgUixD77ey4e7/lvP0GTN7gPAt688JY2SkhKIWogcIBW2+nbsLva9SpdC9cvcFwHdzDn3GzB4ltFRcBJxUyjgFgH8D+hDGJp0PPG5mX3L3WdF+vbcqiLrhxVtG/LfZ/Yn/NkFSxt1fJVTn2ivpWKRFSyn8fsvul5Ry90+Ah9F7reTMrDuhetoo4OC8Cnctva/0t6yEWrhXTbj7h8Cz6H2VCHd/x91fjAo+HAj0IlTFA31GrDhKluLNIPRJzTcGeLvEscjmK/TNqqTLDGC7qBxrrjHAOkIXCEk3vddKzMy6APcCewOHuvubeYc093dsjrurC16JFHGvCp6K3leJc/flhL9D2fGz+oxYYZQsxZsCjDOzUdkN0eSM+9J0DgRJITPbE9iR0O9Y0m0K0AU4NrvBzKqA44DH3L0uqcCkZWbWhzBHlt5rJWJmnYDJhG+8jypQbnoKMMTM9s85rw9wBPo7VjJF3qu484YTPnPofZUwM6shjCP7INqkz4gVxtz1pUU+M6sGXgfWEMa7OHA50BsYq2/k0sXMJhPmDnkVWE4o8HAhsBr4nLsvTjC8imdmE6KHBxL65X+PMIh5kbtPi465EziYUCwgA5wFHA58MepSKSXQ0r0ys/OB0cBUGgs8ZLcd6O7PlD7qymNm1xPuzy8IxTZyzXX3udGH9GeBYYT3VXZS2rHAblE3L2lnRd6rSYQvr18gvN9GE+5VX2Afd59ZwpArmpndT/gs8QahYNSOwHnAYGBvd39XnxErj5KlAqJvda4BvkpoCv87cG7O4D5JCTO7EDie8MGtJ6HE5yPAJdHAWUmQmRX6T2aaux8QHdOD8GHiBKAf4Q/Rj1RdrbRauldmdgSh3/5owge5lYS5RX7u7i+VKMyKZ2azCP/fxZno7pdGxw0Afg0cDXQnfBj/D3d/vQRhCsXdKzM7jfAF0faED9yLCZPVTlSiVFpm9iPgW8BnCFNXfAg8BVyZ+/lPnxEri5IlERERERGRGBqzJCIiIiIiEkPJkoiIiIiISAwlSyIiIiIiIjGULImIiIiIiMRQsiQiIiIiIhJDyZKIiIiIiEgMJUsiIm3MzLyIZVYbvVb36Ho/3oxzD4nOHdcWsbTidbu38Ls5pJTx5MWW/Z18KakYREQkPaqSDkBEpAP6Qt7z+wkT7V6as62ujV6rLnq9OZtx7gvRuW+1USytdQPwh5jt75Q4DhERkVhKlkRE2pi7T899bmZ1wOL87YWYWTd3LyqZ8jCzeFHXjTl3xeae20bmFvs7ERERSYK64YmIJMjM7jSz983sy2Y23czWAJdF+042s2lmtsjMPjGzf5jZCXnnN+mGZ2b/aWb1ZraDmf3NzFaZWcbMLjQzyzmuSTe8KIYnzOzrZvaama02szfN7LCY2E82s3fNbK2ZvR6dM93MHm2j3032Z/uZmV1qZvPMbI2ZTTWzXfKO7WRmF5jZe2a2Ljr2WjOrzjuui5ldbGb/MrO66Hf7sJl9Ju/le5nZDWa21Mw+NrM/mFmfvGudH11nTXTcS2Z2eFv87CIikg5qWRIRSd4g4Hbgl8DbwKpo+3bAncD70fOvALebWVd3/0ML1zTgPuD3wFXAN4ErgFnAHS2cuzPwK+BKYBnwI+A+M9vR3WcDREnBbcA9wLlADXA90B14raUfONLJzJr8HXL3+rxNpwO1wPeAauBy4Ekz28HdV0bH/Bo4D/gv4BFgLCHp/KyZHeTuHiWK9wEHA1cDU4GewAHAYOCDnNf8H+CvwHHAZ4H/JHR5PDP6+b9D+H1eSujO2BPYDRhQ5M8uIiJlQMmSiEjy+gLHufvfcje6+8TsYzPrRPhwPww4i/ixPrk6AVe4+x3R+X8HDgKOp+VkaRDwxZzE6E3gQ+AYQpIBIRF51d2PzYlxJvBcC9fONTFaNmFmvd3905xNVcDB7r422v8KYVzTOcAvzGxw9PgGdz8vOucxM1sO3AR8FXgM+DpwOHCmu9+Yc/37Y2J73N3/I/s4ask6nihZIoz1esXdr8g55+Eif24RESkT6oYnIpK81fmJEoCZ7Wxmd5nZfKAeWA+cBIwu8robP7xHY5tmAMOLOG9GNlGKzp0LLM+ea2bdgN0JrUrkHPc8sKDI2CC0RO0Vs6zOO+7BbKIUvc67wKs0FtL4IiGh+lPeeZMBB/aPnn+N8Hu8tYjY8hOfN4HeZtYvev4ysI+ZXWNm482sRxHXFBGRMqOWJRGR5C3M3xB9KH8cWAr8EMgA6whd3iYUcc0NOV3UsuoI3eRasjRmW+65gwnd/D6OOe6jIq6fNd/dXyniuLhrfgQMiR5nu75tkqi5+xozW5mzfyDwkbuvL+I1838H2YIb2d/BTYS/od8G/i9QZ2YPAedFyaWIiHQAalkSEUmex2zbj5AMnObuk939+Six6FLa0GJ9RIh565h9Ne3wenHXrAHmRY+zic3g3AOi1p4+wJJo02KgJm6cVGu5e4O7/9bd9wS2Av6dcM8mb+m1RUQkPZQsiYikU89ovbEVxMy2Bg5NJpxGUZe418hr4TKzLwLbtMNLHmFmG1vEzGxH4HOEwgoAzxO61/2fvPNOILSATYueP0Zja1Cbcfcl7j6ZUDzis215bRERSZa64YmIpNMzhKp4N5jZZYQWkp8RWnWGJhlY5GfAg2Z2N3ALoVXnEkLXvIYirzE0t2x5joy753a9qwf+ZmaTaKyGtxi4DsDdF5rZdcC5ZraWkBRlq+E9CTwRXedR4CHgN2a2HfAUoVvdAcA90ZiropjZH4BFhHmqFgE7EZK1JmPPRESkfClZEhFJIXefb2bHEEp43wvMJVSiG0EYt5Qod3/IzE4FLiaU2H4X+D6hTPmKIi9zJo3V5XKdA/wm5/mNQGfgd4TxR9OB7+eNyTqfMPbrdMIYosXAzcBPouIWROXDjwEuJBTKOJ9QuOJFWjfWCuBZ4GTgVKA3MJ9Qpr1JdT8RESlfFv0NERER2SJmNgqYSUhQrmqD63UH1gA/dfefb+n1REREWkstSyIi0mpm1pcwKevfCQUUtidMXruclueAEhERKQtKlkREZHOsJ4yd+i2hJPenhEIKF7r7oiQDExERaSvqhiciIiIiIhJDpcNFRERERERiKFkSERERERGJoWRJREREREQkhpIlERERERGRGEqWREREREREYvx/5ocH7YnH0qUAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.rcParams.update({'font.size': 16})\n",
"\n",
"fig, ax1 = plt.subplots(figsize=(12, 6))\n",
"\n",
"ax1.plot(cnn.history['Epoch'].values, \n",
" cnn.history['Loss'].values, 'r-')\n",
"ax1.set_xlabel('Training Epochs')\n",
"ax1.set_ylabel('Loss Function', color='r')\n",
"ax1.tick_params('y', colors='r')\n",
"\n",
"ax2 = ax1.twinx()\n",
"ax2.plot(cnn.history['Epoch'].values, \n",
" cnn.history['Accuracy'].values, 'b-')\n",
"ax2.set_ylabel('Validation Accuracy', \n",
" color='b', labelpad=20)\n",
"ax2.tick_params('y', colors='b')\n",
"\n",
"fig.tight_layout()\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After 30 training epochs the model is at or near convergence for this set of hyperparameters. At this point we could choose to tune our model hyperparameters to see if the model performance can be further improved. This could be done, for example, by performing a grid search while using k-fold cross-validation. \n",
"\n",
"For now, let's continue with the present hyperparameter set and retrain our CNN using the full training set (including the samples we used for obtaining validation accuracy), in preparation for determining the test set accuracy. "
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 01: Training Avg. Loss: 221.667 \n",
"Epoch 02: Training Avg. Loss: 58.428 \n",
"Epoch 03: Training Avg. Loss: 40.410 \n",
"Epoch 04: Training Avg. Loss: 31.101 \n",
"Epoch 05: Training Avg. Loss: 24.913 \n",
"Epoch 06: Training Avg. Loss: 21.663 \n",
"Epoch 07: Training Avg. Loss: 18.848 \n",
"Epoch 08: Training Avg. Loss: 16.286 \n",
"Epoch 09: Training Avg. Loss: 14.257 \n",
"Epoch 10: Training Avg. Loss: 12.718 \n",
"Epoch 11: Training Avg. Loss: 11.156 \n",
"Epoch 12: Training Avg. Loss: 10.002 \n",
"Epoch 13: Training Avg. Loss: 8.672 \n",
"Epoch 14: Training Avg. Loss: 7.631 \n",
"Epoch 15: Training Avg. Loss: 7.675 \n",
"Epoch 16: Training Avg. Loss: 6.329 \n",
"Epoch 17: Training Avg. Loss: 5.787 \n",
"Epoch 18: Training Avg. Loss: 5.339 \n",
"Epoch 19: Training Avg. Loss: 4.423 \n",
"Epoch 20: Training Avg. Loss: 4.304 \n",
"Epoch 21: Training Avg. Loss: 3.702 \n",
"Epoch 22: Training Avg. Loss: 3.802 \n",
"Epoch 23: Training Avg. Loss: 3.539 \n",
"Epoch 24: Training Avg. Loss: 3.044 \n",
"Epoch 25: Training Avg. Loss: 3.175 \n",
"Epoch 26: Training Avg. Loss: 2.381 \n",
"Epoch 27: Training Avg. Loss: 2.432 \n",
"Epoch 28: Training Avg. Loss: 2.199 \n",
"Epoch 29: Training Avg. Loss: 1.899 \n",
"Epoch 30: Training Avg. Loss: 1.920 \n"
]
}
],
"source": [
"cnn.train(training_set=(X_train_standardized, y_train), \n",
" initialize=True)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***\n",
" \n",
"## 5. Testing on Unseen Data\n",
"\n",
"We now obtain our model accuracy on the unseen test data:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test Accuracy: 99.40%\n",
"\n",
"Classification Examples: (predicted class is shown below image)\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA9sAAABvCAYAAAD8OtMiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJztnXm0XEX1/Xc5oTIkDJmAQCCGICAIAjJDJAiBMAgiQQIOgKCwVBDCpPh1QNFFIIgE5Se6mAcRBFmgEIGIzJOMEgIJUyAQhgQHHMD7++Pl1Nv9+lSq+6X7dd/u/Vkr651Ud9+urnOr6t5bu84JRVFACCGEEEIIIYQQjeNdra6AEEIIIYQQQgjRaehmWwghhBBCCCGEaDC62RZCCCGEEEIIIRqMbraFEEIIIYQQQogGo5ttIYQQQgghhBCiwehmWwghhBBCCCGEaDC62RZCCCGEEEIIIRqMbraFEEIIIYQQQogGo5ttIYQQQgghhBCiwbynnjcPGjSoGD58eLPq0jHMnz8fixYtCs38jsGDB8sXNTJr1qxXi6IY0qzjDxo0qBg2bFizDt9RzJ49u+m+GDp0aLMO31E89dRTTfXFCiusoH5RIwPhiyFDmnb4jmLOnDmaL9qEZs8Xuo6qnWZfR8kXtTMQvhgxYkSzDt9RPPHEEzX5oq6b7eHDh+Pcc8/tf626hC996UtN/w75ona23377Z5t5/GHDhuGss85q5ld0DLvssktTfTF06FD85Cc/aeZXdAy77rpr0/vF6aef3syv6Bj22GOPpvpiyJAh+NGPftTMr+gY9t1336b3i+nTpzfzKzqGnXbaqam+0HVU7TT7Okq+qJ1m+2LEiBH45S9/2cyv6Bi22mqrmnwhGbkQQgghhBBCCNFg6lrZFkKIbiOEpu4IqYuiKFpdBSGEEDXQqrlD84QQ7YVWtoUQQgghhBBCiAbTVivbS/sUMPf5/j7t01PCanJt3V9fqK1r413v8p+TLW0f8tr/f//731Ids0w0YwxK+coj1dbml2aNcZ1Cqn1yPuDP1doHur2tc6R8UU8fqbXd5YslU48vUq/X2u7d6ov+zB3eZ+o5ztK2dbf6qlb6ez2gdl0yS3v/0F9aOV5pZVsIIYQQQgghhGgwutkWQgghhBBCCCEaTMtk5LXKCPh9bJvsLPU6YzIBlguwPM3slJzAjttN0hBuyyuvvBIA8N///jeWzZ07N9ozZsyo+sykSZOiPXbs2GiPHz8eQH2+6KZ2N3IycX491y9y5HyRolPk5fW0Va6tPb+kfMWYD1Ltb6+/8847VWWp+nViv8nJxFNt/e53v3uJrzOeL7jdrZzLvM93Oqn2s7ZO9Yul9cXbb78dbSv3+kpfu5PJycRTbZ27zmK86yhv7shthSk7/Z0var2mreW7cltdar2O6vT5IoXni9zcnSM3d3fK9VK91LpVYiDv9XI0oy9oZVsIIYQQQgghhGgwutkWQgghhBBCCCEaTNNl5PVEnTN5Gdtc9p739Fb3fe97X9XrKcmHyf1YBu3ZLAv05JqdKrnxJDWnnnpqtG+++WYAfvsDwJAhQwAA733ve2PZn//852g///zz0f7EJz4BAFh++eVjWX980Unt78mXUv3C3sttzX6xcn6dP//iiy9Ge8cddwQATJkyJZbtvffe0TZfsGzTk86WVR5V69jkjUtsc/tz+5xzzjkAgMsvvzyWbbTRRtGeNm1atFdccUUA6THqP//5T9XxPZv7RdnHq5yUjNvdbPYPj1HWH/gz/F5uH2tLa3PA9wuXeX2kjG2ewpPwpfqFtbXX/lzOr6f6orWr1xcA4N///nfFX/4M0D2+aOR8kfKFdx3FvrB2T80Xue167U6t8wXP557tjVtA/prWG6O4rZf2Oqrs84VHyhfe3O31i1ruL/72t78BAObPnx/LvDFqlVVWiWWXXXZZtEeOHAkAWG211WLZmDFj0j+qTcltZcltK/LaP/V67l4v1y9y17RMo/qCVraFEEIIIYQQQogG07SV7dwG+Fqf8r3//e+vKgOAZZZZpqos9bTDnmbw06Z//etfVTY/HfdWLfgJSNmfAnpP/H784x/HMl6ZXmGFFQAAH/jAB2LZJptsEu2tt94aALBw4cJYdt9990Wb22f27NkAeldVgcp2N1+wf9gX5oNcoKh2JxfIKbVCZ+c99wu27XX72/dYrDIYPXo0AGDTTTeNZbbCCgD//Oc/AVT6gvuQ54vcU8JWU+u4BOSfuHpj1Jtvvhnt22+/HQCwzjrrxDI+l+fNmxdtCyLojUtA3hf2G1JBvcoS5DHni9Rq6Zw5cwAARx11VCy7++67o20+SvULLwAXt/XMmTOjPWzYMADAcsstF8u8VfAy9QuPnOrMG5eA3rbm+eKDH/xgtK085Yt77rkn2hMmTAAAfPe7341le+21V7S9fvHWW29F2/ziBVUrE6ngQN58wb//hBNOAABsu+22sWzy5MnRHjp0KID6rqNSY5R97xtvvBHLbAwEgC233LLq+O3ui9xqtncdlVPX8Hnv9Rse13Ir29z+3nUUl+Xmbm+Vu93nCybXL7hdzfaunQBfffPUU09F+4477oj2/fffDwB47LHHYtmiRYui/Y9//ANArwIUqAwwbP2G+wLPN+1Of+71ckonr1+kVrZzqrT+3Os1I8ijVraFEEIIIYQQQogGo5ttIYQQQgghhBCiwQxonm1e+jdJAEsHPEkHS9HY9gLe5KQFLBfwPpeShudyt5VFcpOSaD7zzDMAKqXjyy67bLQ322wzAJVB01ZeeeWq9/IxTzvttGibxJPfywHSagl80JeUL9rdB9655gUB5H7h9QGWZbJtn0u1KcucN9hgAwDAzjvvHMtM8tT3cx4m1cnlXW13PLky0NuGKdmfncv8O3/2s59Fe9SoUQAqz3U+PgdEsa0aPAZ6vsjJGnmMK9NWFy+4kCfH5PbnfvH4448DqNwGwbZ9rp75gn3xyCOPRPt3v/sdgF6Jbl9y80VZfJEaozxJPo9BJq/nOYRt+xxLCfm8PfPMM6O9xhprAACmTp0ayw488MBo2zHqmbvLlIc75ws7n1kmP2nSpKrjfPzjH4+2bR8C8vOF1y/Y7+xDmw/YP7xt6aKLLgJQOe6VyRdGKuiWN3d74xX78tprr432q6++CgA4/vjjY1kuiKMnjeZ69featiy+yG11yUn2eYxnubEFBba/QLqtzBcW6AyonJvMr/Y3Ve+ytHmK1P2FFzAzt+2I39ufMSq13Sx3TVtrAOb++Eor20IIIYQQQgghRIPRzbYQQgghhBBCCNFgGioj9ySOvGzPkSlNnsHSi5VWWinalu93+PDhsWzw4MHR9vKqpiQzOUlNLjeblecknO1KTqJp0mKW1HCEapPwcTRFL2+nRWUEgAULFkSbj7v55psDqJR5MCbPYV94dll9YeTkstxmnmScZZmeZCYlHb/mmmuive+++1Z93pNK1eOLdvRLLv9jPdJlbvcZM2YAAB5++OFYxnnMbTzjzzMmfeY6WHTgvnat+SP5t5QxArYnPwP8fsG2RYLl+cLbdpTKAuDVgV/naM623YZf57p6+YbL5Bf7Xbnosdy+HJndbC5jX3k5bHmr0euvvx7tQYMGAaiMoM3XATa2pfLTe3lVO2l7hUXVPfnkk2MZ/6YjjzwSAHDMMcfEMh7v+3MdxfDrV155JQDgtddei2Vcr7XWWgtA+291qSdfsBftOrfV5de//nUsY+kyv9fgPpi7pmW8aONeVpeybPVKUc/czWOQtx3PMhsAwB//+MeK4/T9Lm43+172Bbe7bYXh7WR///vfq76XI2S3O941H5d5c0dqy7D5ILU9xdqF/cPXTnzfYeM8Zzvi/Ob2vak82941rfJsCyGEEEIIIYQQbcpSr2zXs2p0zjnnRNvy0D3xxBOxjFe2H3zwQQDAiBEjYtmYMWOizU9ODC9PIJfzkz0OPmTH4kBRHFynHVfr6sHzBduWJ/vqq6+OZRwAzfySC2T2hz/8Idp/+9vf3Lp4T1K9ICP8lLKW1ah2xmv3/gblsiey/OTPC6LCT+s4SA3nQN1qq62WWFcvV2VqBaQdqTVHaj2+4Cey06dPB1D55JRXvs0XvHrBfvn9738f7RtuuAFAZb+bMmVKtG2FL9UH290XHrm8qDlfzJ49O9q2sn3iiSfGMi+ISi2ryvZerh8/VbeAknysTvKF1y9yOWq9nNqpVSFrX17J4eCb3txhKhzAPy9S42HZfZFb2X7yyScBAH/5y19iGV9HHX744QDyQYJ4paee+rEK4cILLwQA7LnnnrHM8qQD7bNy3V9yvvACpPE5fv7551cdk/uQtc/ll18eyz796U9H21MDpoJSeddRbJcdbzXV8wu3P1+fbrPNNgCA9dZbL5Z5QWl5vuFjWdBgoDfgIAf+Y/WNfY5zP/P9h+XZ9pQHQHn6Ter+wnzBY5CX05xX+zmP+W233QagMnd57l6Px0O+/rLrs1VXXTWWHXDAAZlf1hi0si2EEEIIIYQQQjQY3WwLIYQQQgghhBANpul5tnmJ/6STTor2/PnzAQAf/ehHYxkHcnr55ZcBVEoFH3300WibrNKO0xeWW5pkwT4DVAYyeuWVV6o+w/KdWnOvtTs5yQ0HQGM5rBcAguXgt956K4DKoAUsS/vwhz8cbZPJ9je/Zrf4wgtIBPSeo6lgHOYX9g/nrWW/mvy5v22Zy2HbjuQCe3i+4HHh29/+drS9wGcsN/bkYynJuY1BLKHlfLWcj3VJtHv750gF2zMfvPTSS7GMgz5tuOGGACrlxtwHTKKXCgiUkoYaFjAH8Nu4nhy2ZaEeWWA9Y5T54umnn45llru873FNzjl27Niqz9dCGceoHHydZMH6uP1PP/30aHNQJmNp+wUHQDv66KOjbe26ww47xDKWSdvY2K6+8LYc5MpyW2F43Lb5gNuE5wa7ZuLtfDfeeGO0P//5z0fbtoCl8g175MatMpGbu228+f73vx/LOGjvdtttV/V5npu/+c1vAqico3luXmGFFaLtzRd8/Wt9i2XoufmiHf2S25JTzzUtbz8xmfhdd90Vy/g6ytqS799svgeANddcM9oWfJC3HD/00EPRNqn6nXfeGct46964ceMANKf9tbIthBBCCCGEEEI0GN1sCyGEEEIIIYQQDabpMnJm4403jraXL5htk2iyzIajKpvkmctS0VPtuCwXOOKII6JtMgWWUXuyqpS0oB0lHzly0We9iK7cJn/961+jPW3aNACVUf+GDRsW7f3226/qWCnJjJf/ke2ySG7qIecLL3osyzLZtv7CUj+T6QDARhttVHXcVFt7knQvSmbZ2z+F+WXWrFmx7IUXXoi29REeoyyyKNAbcXTSpEmxbPvtt482Ryy1LTLnnXdeLOO2Npmtyd+A7uwXv/jFL2IZy4lPOeWUivcBlbI985GXHxWonC+8z99yyy1V38v+8SKilr39GU8uy9srvPnCy18K9Pri4osvjmUcaZYjm+++++4A0mOUl38+t62m7EydOjXaM2bMAFCZV9ayiwD+3M1tZXaqX3gSWc4gw+Phpz71KQC9UkygUnrb6f3CbP7NPG5Yf+Go1Zzn2bYE8PnLWwY4mvkmm2wCoLKveNdRXl/oa3ufLyN8Xv/kJz8B0ButHwC+9rWvRXvttdcGkO4XNh/wNS3L/72xP5Wb2b4jNTd3Yr9gbE4+99xzYxlvObX24bmbo8Rb5HDe2pvKPmGS8cMOOyyWnXbaadG+5557qupn1w4A8LGPfQxA5RbBRl1TaWVbCCGEEEIIIYRoMAO6ss3kcj/bEwR+es5PBO0p1FprrRXL+L3ekw9+Issrf/aUy3Lv8fd3E6lcsdYW/BTw4YcfjrY9feUnTNtuu220R40aVXXcVPCg3BPZspM7r3IqA8NbzQZ6V904zyDD6o6cL6yPpXzh/ZZ291XuibKXQ/bYY4+NZewXW5Xg1WxuX8s3y4FteCWDgxqNGDECAHDNNddUfT/Q+/T1ueeei2W77rrrEn9fmfDqzb6wAGW2kgcAm2++ebRXX311AJXnKq9M27mcGuO88ksvvTSW8Xm9xRZbAKh8+s0rIJ0UiMgjp77x5gv2hY1RM2fOjGXsC+4jnGve8MYoLyAR14Upky+8unpzA+eN5WsfzxesCLF285QLfT9nwb6uuOKKqs8DvSuHnC+37L5IzQ1emfll3rx5sYznBhuvjjvuOPd1W1W77LLLYpkFzgQqV7ktCB7PTUyt11Ht3v4prN48rlx11VXRtrGFAytOnDgx2naOe4EbAX++YLz5or/XUX1/UyfA44LNoxwscOjQodG2sWuPPfaIZazgM1K+4D5oYw+3L+fRvummmwBU3v+xX5qJVraFEEIIIYQQQogGo5ttIYQQQgghhBCiwSy1jDwls8nJAj0pmifTScmQvNxtLCNn22S2Z599dixjyYgF8OIACCzvKTs5X1hb5nxhgdAA4I477oi2yWFZpsPSDfaRFzzHk5G3uxy5HlLBZwyvX3gBh4B8ICw771nKtuKKK0b78MMPr6pLKtiaJ3/qJKmTh9fWLN9jmbi9vtlmm8UylgiaHJbl4qnxyiTJBx98cCz7xje+EW2rAwfz4ICTFtwxlwuzncidS56MnOVpPMYYqXHDzvWUL9g2Kdr111/v1vWLX/xi1fHL3i9yOcE9yTiPUZ7ELzV3P/XUUwAq859yDlXeLmY5VFMB0DpxjMpJl7mtbQzhQFzf+ta3or3SSisBqJRosvTW4C0RHPz0sccei7Ztw+McuOPHj4+2J10uky+srv0dQ71gdNxHLICcdx0L9G5jvO+++2IZX6eyj6y/eLml2c7J+MsOB3+94IILov3hD38YADB9+vRYxtf3Ruo6qp77i/5c0+auydvRV7lxiXnkkUeibYEwOTc5B1A++eSTAVQGqObx3uC+4t3fAcBee+0FoPI6iYNoW3/ifsVjo12zNUNarpVtIYQQQgghhBCiwehmWwghhBBCCCGEaDADGo3ck6LlImAzXl7PlMyDj3vXXXcBqMx/uNxyy0XbouHVI2tsR5lHPXj5IVO+sHzBHM2dGTlyJADgwAMPjGWcM92LVFtP/sdc/cuI1/5A7dsrGD7vX3zxRQCVMn+ODG/yKv6OlOwvl/+x3X1Qqyww5QsbY1Jy2Q033BAAcPzxx8ey5ZdfPto2NtUyRlldLX8qHx+ozGfbyXD7c4Rvk1ayLyZMmBBt7xzl95p8vBZfWCTbV199NZaNHj062uYXGxdTv6Hd+0eOXL/o7xj19NNPA6iMSMvzMW/FyG118XLYer+h7L5gJk+eHG2WfBv33ntvtK3db7jhhljGEk2bp1nGz3M39xfL6cxyz0MOOSTaubm77L6odTvezTff7L5+9913AwDGjBlT9Rmg11ccgZz9wu+1yOXsH5bGdku/4GtS/i3rr78+gMr2q+f+wuyUdLyea9pc1pNOIfWbbPsDtxmft8888wyAyi0Bc+bMibbJ/21LDFDZR1566aVo29jFc/eiRYuibT7i7YCcLcZ83IztF1rZFkIIIYQQQgghGkzTV7ZzK3iM9+Qnlf/RWzXi1xcsWBDtSy65pOr4tikf6H1Syyspnf7kyVMZpJ78TZ06FUClMsCecgPA3nvvDaDyaRH7go/lBR6o54lfJ/qF28oLoMZ4q7X8xNWeCPLrH/nIR6LNAaJq9UWqDy+prEzk+gXDbWa5TlPHsie6qTHK6xcplYf3dPVnP/tZtC0oUif5gldq7En1oYceWvPn+Vw3m8tSvrBc5ly2zjrrVH1Hql+U3QdGql9Yu6XmC2+M4nZ/9tlnAVTOF2ussUa0OWCNl8M2V9fcdUYZ4d+x7rrrRvu3v/0tgMrVGw48Z0Gjhg8fHsssiBDQG7SIgxdZoC4AOOKII6JtgYhs1RCoXG2y64Oy94XU9YjX7z11xyc/+clY9uijj0bbVmHfeOONWPbyyy9H++GHH646Jqs/+PrruuuuA1B5LnAAtVrn7rJz4403RpvbzZR91j8AYNy4cdEeMWJE1bH4Oip3f9Gfa9pOv7bl+nPw1i233BJAb2BMAJg/f360zzjjjKpjsS9sbOJ7Dp5P2O8LFy4EkO7D1jd5NZv7jRc8slF0xkwkhBBCCCGEEEK0EbrZFkIIIYQQQgghGkzTZOQ5yY0HyyZzn/eCtDCPP/54tE0awPInC+rFr5ddxpGiVl+w9ILbb/bs2VWfYZnIzjvvnDwm4Etjaw2m0tfuFOr5fSxT8iScLG+y4EMsf2JZoJf/MZcXNRUIKZcDtiy50lP94sorrwTgtxngy2W9YGq19As7Lgc3uv/++6Nt/uTjH3nkkekfVVK4rThQ03rrrQegMu8vyyotMF3KF7n5gnMHm0STJWUbbbRR1Wfq6cNlCq7p5dH2trekpJTefDN37txoX3311QAqg6KxjJnlgraVIDVG1RpQMuWrdveFkRqjrI9w0LINNtgg2gcddFDV59mXJqFk2SYH/uNgaib1t2MC9fki55ey+8LKecuJ5e0FemW0LM3n160/bL311rFs//33j/Zpp50WbetP119/fSyzPN5cl3qCGJaRN998M9o8htiWh1/96lex7Kqrroq2baXgLXY8n6y99toAKuXm3JYs/7e+l9sC2E3bjljmfeKJJwKobJ+bbrop2hbkcfDgwbFs1KhR0bY+wvmy7Z4EqJwbbLzicWvixInRNr/zWMN5upuJVraFEEIIIYQQQogGo5ttIYQQQgghhBCiwbQsGrmXm86L6peKDmzv5WPy52+99daq8v322889VqfIOHLkpCt///vfo33hhRdG22RlLDVba621qo7l+Q/w5bIpiXGn5H9MkZN3ef2CozLb6yzJYfmTyZA5iilH+mV5jeeLeiLDl0X2lyIXsdWkTuwLzlhgbZmKUmrtym3Ox2fpsuVHP+ecc5ZY5xVXXDHaOdlamfB8wVFCP/ShDwGozGHLMnqLUs7bJ1iiadJkzslpUbH72uaXlLys0/uFUc/czee4weenRYkFeucTPibnl+fxLjdGefXzKKtPvPmCsd/F7cO+8OaL3HXU5ZdfHm0e7774xS8CqOyX9WRwKYsP6tkS4vULbp8pU6ZE+4c//CGAynGf37vbbrsBAHbfffdYxn6zqM5Ab2Tzu+66K5Ztv/320bbcxJ2KtfvnPve5WHbNNddE23zBYwmfq7/5zW8AANdee20s421LNnewfxjuY/ae0aNHx7IvfOELVXVlvOwNZekfKXL3Fzw377nnntG27Q88LnmR4TmyvEXuByp9Ycf4+te/HstsCxrQuzXsrbfeyv6eRqOVbSGEEEIIIYQQosHoZlsIIYQQQgghhGgwTY9GnpM/peSyJouqRyo5c+bMaJvMBuiNzrnqqqvGMi95edllHCly0mz73Ryt8YEHHoi2yTjGjRsXy3bZZZdoexLCFObXXFt7MptaPtfu5KTLXr/wXud24Iikc+bMAQBsvvnmscyTFQLyRU7Sbz7gcYnfa+W5qOspaT7LNc8//3wAlZGAmdVXXx0AcMIJJ8QyjkBs41lZfZLrF1/5ylcAVEaqvuWWW6JtckqWqrGU0iL98vYXlqrNnz8/2taW3AcnTJgQbfO715f6lpeRXBR9+32p7RHe3H3xxRdH27a9DBo0KJbtscce0fa2ltUzRqWyB5QFrw/kJJopSX+t2T84yr9lYQAqt2KYXDY3B3TrfOHN3WuuuWa0jzvuOACV16krr7xytMePHw8gPZZwlHKT0fIYeNlll0X785//fFX9O9EXhxxySCxj+f0ZZ5wBoHIOeO2116Jd63ZGL0NS33Kbs+fNmxfL2K877bRT9veUmf5c03Ifysnob7/9dgCVkn+Gj3vMMccAAMaMGRPL+Pqt73cCAzd3a2VbCCGEEEIIIYRoMAOaZ7vWQAFsp55S25MRzrd2wQUXRJufaHFgNA/vyYr3tKOsTwNrzbl46aWXRtsLqMJBH/h17+m59+QqRS5YRO7pebvjtXs9Oee9tuC+wMGdbFWOVyRybZVTPOSe/JXJF/1ZNfLyxLOdCtLorRCeeuqp0Wa/2XtSeWfHjh0LAFh//fVjGQf5KJMPjHp8MXLkSADA9773vVjGwc4swByvVnOuTwt+8/GPfzyWsV/OPPPMaF9xxRVVr/OKuQVOy81dZfKJ1+65+SL1+61fvPHGG7GM892aOoFz2LLqrB688bDsvmD6cx3lrebnxihebeUxjoNyDRs2DEBayeZdB3TSdZQ3Rud8wb/fcjdbsEegcryy46fah/vjDjvsAKByZZuDRnEQNq9eZfWBYe3O16EcoOzcc88FUDluP/PMM1Wfv+iii2KZ5UEHeq+j+Pgpla13Djz55JPRNsVCpys+6gmQ6CkG+PW777472r/4xS+qjsW+YPWIXSel2tRTcw7UfKGVbSGEEEIIIYQQosHoZlsIIYQQQgghhGgwSy0jT0kHPGmBlzM7lUfbZAJeGdC76f28885z68LyJ5OoeQHYgF5ZFcurUlKoslCPRNNsljR5ckzOO8sB5jzpRSpnnhd4gvN7W7AJDmbB2wO84EWMBcxgCWk7UWu/SMmXvNfvvPPOaFsQm+222849vhesKxdcKLWVo9YAa+1KTv5kbZzylQVhZF9YLlUAePPNNwFU5u/MBXxkiSb3t1NOOQVAZQC1TvJFTqLpnfcc7Mykmfy6l6szNd/wsewYfN4/99xz0R48eDAAP2c9UH65bG6+yI1R1u6PPvpoLGM5p43NHBQtN0YxXjAwL6haX7uMeL8/J/n3fJF63crvv//+WLbCCitE+zOf+cwS6+Ll9/YCtAHl7Be59vfKU23tXdN6dmq+YSx3MM/zHODWchJz0DDvOrZMvmBqHaO4/UxiDPT6gnOTP//889G2rS4TJ06MZTvuuGO0OSjtTTfdBCB9nWR2J80X/bm/yI1BHKT05z//ebStXbyApwBwxBFHVB03d6+X8pUCpAkhhBBCCCGEECVCN9tCCCGEEEIIIUSDaVo0ciO1LO9FE2SpmcmfPLkBAEybNg0AsGjRoljGUen22WefqjqkcnqbXLOTZB4eKXme2Sy9Njky0Cvf4By//LrZ7D9miy22iLbJc+bOnRvLOK9UrWj2AAARCElEQVSnRbC1XKwA8K9//Sva5qtUhGzL3XrAAQe4dWkX+tsvzGZZK5/LJnFl//CxPOlxSrpsNpflJDdl3HKR8sXkyZMBANOnT49lLAk/6qijAFRGfmep00orrQSg0hf8Xdyu1m68PWPfffetei/72vNlGdufyY1RqX5hY1dO2lxLRFnrDzzuDB06NNq2PSDlC0+63O5zRy6qck5G7vmC28HGJaBXsn/ggQfGstwY5c3XQG9/4X7jSQg7KQ96Pb4wO9UvHnjgAQCV5zr7yiKQA73tmpovvNdz80UZfZGLFp67ps3JzFNjlNeWvBXj17/+dbTPOussAMAmm2wSy9ivZd92lLsm9+YLb5skZ6fgPM42Z996662xjLMr8LYLb+5mmXOuX5R9zjbqmS/YF1Zu2/L6vtd8sfzyy8eyk046Kdo8N1tbc5t6Y1RKZt5MX2hlWwghhBBCCCGEaDBNW9n2VpO9J3P8BMR78sFPQDh34Lx58wBUPjnkPNC82mRPnDgvLT/JtddTK3hlf/JUqy+22mqrWPbII49E29rSW80G/OBD/OTo5ptvjrY9HXz11Vdjma0UAb2+SD2x3GmnnQBU5qrk91rgkHbF84W36pLqF2bbigRQ+RR1gw02AACsscYaVcfs+13mo1S/MDu3mlrWp+O5vLDjxo0DULliwG1h/YIDCvHKt63wsS/56Tc/qV1ttdUAVOZ7Zr+bLzp1jPJ8wSsBdo7lAkGlgjx6q0XcftyuNh7xuMRzj/mQzwUvKFSZfOKNEfXM3d5q6r333hvLeFXNcsWbyqnv93t+4cCAbJtfvNUL/i1lHaOs3ql+YeXsC68PpPrCDTfcAKBS1bbNNttEm31hgUxfeeWVWMaf6/T5wvNFTjHhjVepFT4vQFpqjLc2NiUfAEyaNCnaP/jBDwAAZ599diw7+uijq+pSVjxf9Of+YsSIEbFswoQJ0b7jjjuqPv/QQw9Fm8cbG4823HDDWMaBBb35opv6hTdGeQExLagf4F9HsYpj+PDh0eZ2Ndu7vwN8taY3nirPthBCCCGEEEIIUQJ0sy2EEEIIIYQQQjSYpZaRpzbFe1I0L8gJy8/4vSbzYMkY57C1pf+DDz44lrEkhGUEJjVjKaAnC+ykwB45WSD7wtr6q1/9aiy77bbbom2SDk/aAQDPPvssgEq5OEuTFy5cGG0LSMESQs6zvdlmmwGoDHrQScHqPF+wzMV84QUcAnr72O233+4ef/z48VXH5/Oa+4X1rZREs1ZZYNnlsqmAPybz5nHnvvvui7YFVOFxj4/vBUbhgI5f/vKXo21bOFK+yI1ROflTu/eXnC/sXOS+wH3Ekw16x/ekmADw4osvRnvBggUAKueIXL/w/NLubZ7C2sgLRAb4vuC2MLmmzQt9sTEuNR/x3GGBMrnfeL7gca2TJJpev8hdR3lBuVJ4W4k4OOnMmTOj/dOf/hRAb+BHoDdIJNCZctl65gtPsp/bIuZ9V2ru9qTL7Lett9462iuvvDIA4LrrrotlHDTYghSWfb5Y2vsL9hW3j7UrB+3iPNAcCHWXXXYBUClDZ794c3fZAzbm7i/4XPVyX/PnLfCy5x+gd0ukbevre3y2rd157vZ8kQtoyjTKP1rZFkIIIYQQQgghGoxutoUQQgghhBBCiAYzoNHIWYrm4UWws2iZADB79uyqz6yyyirRZjkyy8pMRpCKUGd1zEnHy0pOuuzBEUlN8s2yQY4muPHGGwOojBbIx/eiXXOZF2E55Ysy+sWTWbOkJgf/ZosCz+1j0X2B3ryaLKPxtlSwzWVelMZURNQyycc9PFkgn7cm91t99dVj2dixY6O95ZZbAqgco1jeb7Kn3XffPZaxBJYjyVrk63p8UfbxyotAzf3Cy7PNeOdiSormzUc8Bp1xxhnRtv5y+OGHu+/tTz7hdsfzBZ9/uTzcXlTgkSNHxrI//elP0TYJJsuVve0t/B5POg70bwtYu5OTLufmbu/zLHvlqMyvvfYagErJP/cFjsi/2267AQD233//WMb9IjdflJ3+XNN6vuA2YV94UZ297RtAb7tz+/N4N3XqVACV12SXXHJJtE888cQl1rvdsbbiflHPfOG1NV/TTp48GUBlPu177rkn2hMnTqz6HI9R3tzdSddOjOcLxvPF008/He1Zs2YBqLwe4mxHJu/nvpDadmQ2v5dtb4zyfouikQshhBBCCCGEECWgoSvbXrC0VEAaK+cyfnL34IMPAgB+9KMfxTJ+QmHfZfm2+36/9zSjv8GFvOOXkdQT/5wvbLXNywmZOn4qsEd/fFH2dmdyq9zeSgY/ubMnflOmTIll/HTc8penglV4+WjrCYBWxieyqSCO3koF4/UL9sWQIUMAAIceemgsO+yww5Z4fMvNDfi5nVP9wux6AquUqd94T8frGaOsX7D6xsslmxqXzJdAbz7aTTfdNJbxfOKtopeprXPkfGHl7AtWZNgq6t577x3LWJVm88gLL7wQy7xcqGynVjU6XZXmrcD1Z75IqdL22msvAMBzzz0Xy7bbbrto77nnntHmYFNLqktqjiiLX3LzRW4M5nPRu47idvTyDad87dmp+cKC2PEYNmPGjGh/9rOfBQCMGjVqib+l3ann/sJTW+auaVdbbbVos0LNu45Kqc7KHjDTo557PXsvl5188snRtrGHV7M58LXNJ3Zt2/dY3pyeuhfxxqaB8otWtoUQQgghhBBCiAajm20hhBBCCCGEEKLBND1AWioHrRc0gJf+Lb/j3LlzYxkfy3Jqc/7N5ZZbLtqe9DUV0KXW39JJ5IJFeFIoltnUEzDHK6/lvZ1MTmrHr3t5snO+SEm/++OLMkrHU3jyJ29c4vL++sILtpFqa0+CmfOVR9n7jxeoi8tT+YZNFtjfMWratGlV5SwVzM0nnUjKF965yhJNCxTEEs1jjz226rivvPKKe/zc9gnvvOgmX+TGaG+rRapf2PaJ73znO+7xvXbP1aWTfJGbL3LXtF5gO0+unPrO1HyQk7Sb/YMf/CCWcWA723655pprLrEuZSLXLxo5d+cC33XqNkiPWu/1eNzmgIwWpJGDXfNWltdff73qmPWMh/X4opm+0sq2EEIIIYQQQgjRYHSzLYQQQgghhBBCNJimyciNVGRHI7XcbxI+juY4evToaJ999tkAgGWXXbbqM0uqQ6117SZqjbKZymPovTfVlp0oNVtavFyPqVyl1u5eX0qRk9GkfNFJ8nEPT/7EeBLWZvliaaNkdmJ/ykn1cr6oZ4zyXi97VOVG4v3mVJTXWvtFLfK+Ts9O0R9y56on/R6I+aLT/ZKbL3LbsrzP5ba61NPWXhnnV7/66quzx+0UvEwgSztGpco7vS1rJRfFn7Fo+EBvTviDDjoolvF5a/d1jZSDt8JnWtkWQgghhBBCCCEaTNNXtpl6njbst99+FX9TdHugs0aip3Wtp9NXk9uR1LnuBcRJ5eQeKLq1X+by2YqBw1MnadxqDVKKDTz1zBet8ovOh2rawS/dQk55sc8++1TZvBre37m9nf2qlW0hhBBCCCGEEKLB6GZbCCGEEEIIIYRoMKGeZfcQwgIAz2bfKNYsimJIM79AvqiLpvpDvqgL+aJ9kC/aB/mifZAv2gf5on2QL9oH+aJ9qMkXdd1sCyGEEEIIIYQQIo9k5EIIIYQQQgghRIPRzbYQQgghhBBCCNFgSnezHUK4NYRQJP79vtX16yZCCJ8OIfwmhPBsCOGtEMKsEMIPQwjLt7pu3UgIYfUQwlkhhDtDCP9c3CdGtbpe3UgIYWQI4coQwqIQwpshhKtCCGu0ul4CCCH8fnHf+H6r69JtaIxqX9QvWksIYVwI4c+Lr6VeDyFcGEIY1up6dRshhB0S9xcLW123bqRT+kXpbrYBfAXAln3+Hb34tWtbVaku5RgA7wA4EcAuAM4B8GUAN4UQynhulZ0PAfgMgDcA3NbiunQtIYQPArgZwLoAPgfgQABjANwSQli2lXXrdkII+wPYqNX16GI0RrUh6hetJYSwLYAbASwEsA+ArwHYDsAfQwjLtLJuXcxXUXmfMb611ek+OqlfvKfVFaiXoige71sWQjgUwH8AXDbwNepqdi+KYgH9f2YI4XUA5wPYAT03HGLg+FNRFMMAIIRwCIBPtrg+3cqhANYGMLYoiqcAIITwMIDZAA4DcHoL69a1hBAGAzgDwFEALmlxdboVjVFthvpFW/Bt9ER/3qsoircBIITwBIB7ABwMYHoL69at/LUoirtaXYkup2P6RelXH0MIHwCwL4DfFUXxeqvr0030udE27l38d7WBrIsAiqL4X6vrIAAAewC4y260AaAoirkAbgewZ8tqJX4M4LGiKC5tdUW6FY1RbYn6RevZAsBNdkMBAEVR3AvgNQCfalmthGgtHdMvSn+zDWBvAMujZzVVtJ7tF//9a0trIUTrWB/Ao075YwDWG+C6CAAhhG0AHISebUhCCKhftBHvoEed2Zd/A9hggOsierg4hPBOCOG1EMIlirnSEjqmX5RORu5wEIBXANzQ6op0OyGE1QB8F8CMoijua3V9hGgRK6FnT2pfXgew4gDXpesJIbwXwM8BnFYUxaxW10eIdkD9oq2YhZ5VvEgIYU0AIwD8tyU16l4WAZgKYCaANwFsjJ64RHeGEDYuiuKVVlauy+iYflHqle0QwqroCVpwMcsMxMATQlgOwDUA3gbwhRZXR4hWUzhlYcBrIQDgOAAfAHBKqysiRBuhftE+nAlg8xDC90MIQ0MI6wK4EMD/Fv8TA0RRFA8WRXFMURS/K4piZlEU09ATAHgYeoKmiYGjY/pFqW+2AUxGz2+QhLyFhBDej55I8GsD2LkoihdaXCUhWskb6Fnd7suK8Fe8RZNYLP07CcC3ACwTQhi8OCAU6P/vbl0NhRh41C/ai6IoLgbwfQDfAPAygMcBzANwPYCXWlg1AaAoigcAPAlgs1bXpZvopH5R9pvtgwA8VBTFQ62uSLeyWIr2GwCbA9i1KIpHWlwlIVrNY+jZt92X9dAzWYiBY20A7wdwEXoedNg/oCd14RsAPtKaqgnRMtQv2oyiKL4FYBUAGwIYURTF/uhJGfnnllZMGAG+Yk00kU7pF6Xdsx1C2BQ9F7RH594rmsPiXNoXA9gRwG5KkyAEgB6Vx2khhLWLopgDACGEUQC2BnB8C+vVjfwFwDin/Bb03GicB+Ap53UhOhn1izakKIp/AHgEAEIIuwBYFz0pjkQLWXy/sQ6AK1pdl26kE/pFaW+20bOq/TaUF7KVnI2etGunAPhHCIEDGbwgOfnAE0L49GLzY4v/TgghLACwoCiKmS2qVrfx/wAcCeCaEMI30fM0/HsAnkdPQCIxQBRFsRDArX3LQwgA8GxRFFWvieaiMar1qF+0FyGEjQFMAPDA4qJtABwL4MdFUdzRsop1ISGEiwHMRY8vFqInQNoJ6JEvn9XCqnUdndQvQlGUTxWxWLr8Inpy2e7e6vp0KyGEZwCsmXj5O0VR/N/A1UYAQAgh1aFnFkWxw0DWpZtZvCfyDAA7oUd+9kcAXy+K4plW1kv0sLifnFIUxTdbXZduQ2NU+6J+0RpCCOuj50HsBgCWQU/q1LOKovhVSyvWhYQQTgCwP3qubT8IYD56sh19uyiKUu0TLjud1C9KebMthBBCCCGEEEK0M2UPkCaEEEIIIYQQQrQdutkWQgghhBBCCCEajG62hRBCCCGEEEKIBqObbSGEEEIIIYQQosHoZlsIIYQQQgghhGgwutkWQgghhBBCCCEajG62hRBCCCGEEEKIBqObbSGEEEIIIYQQosHoZlsIIYQQQgghhGgw/x8ZqDGbSNSB2AAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# obtain predictions for test set\n",
"y_test_preds = cnn.predict(X_test_standardized)\n",
"\n",
"# print model's accuracy\n",
"print('Test Accuracy: %.2f%%' % \\\n",
" (100*np.sum(y_test == y_test_preds)/len(y_test)))\n",
"\n",
"# let's look at some examples\n",
"print('\\nClassification Examples: (predicted class is shown below image)')\n",
"fix, ax = plt.subplots(nrows=1, ncols=10, \n",
" sharex=True, sharey=True, \n",
" figsize=(14, 8))\n",
"ax = ax.flatten()\n",
"for i in range(10):\n",
" img = X_test_standardized[:10][i].reshape(28, 28)\n",
" ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
" ax[i].set_xlabel(y_test_preds[:10][i])\n",
"ax[0].set_xticks([])\n",
"ax[0].set_yticks([])\n",
"plt.tight_layout()\n",
"plt.show()\n",
"\n",
"# print(cnn.predict(X_test_standardized[:10, :]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's examine some of the cases where our model classified the data incorrectly:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Misclassified Digits:\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA9oAAABuCAYAAADYpGu5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJztnXn8XdO5/z/LTGUgMqEa85iYiSQ3iKGRxnAFEVw1RCteTc11abV6kfai1FCaKP1RSWooSlFTisQtQmOImoIYQkJISCg17N8fez/rPOecZ+91zjfnfM/Z53zer9f3lZV19rie/ay191qf9SwXRREIIYQQQgghhBBSG5Zr9AUQQgghhBBCCCGtBD+0CSGEEEIIIYSQGsIPbUIIIYQQQgghpIbwQ5sQQgghhBBCCKkh/NAmhBBCCCGEEEJqCD+0CSGEEEIIIYSQGsIPbUIIIYQQQgghpIbwQ5sQQgghhBBCCKkh/NAmhBBCCCGEEEJqyArVbNytW7eod+/e9bqWlmDBggX46KOPXL3P071796hPnz71Pk2umT9/PhYvXtwptujbt2+9T5N7XnzxxYVRFPWs5znoF5Xx0ksv1d0WbC8q45VXXqFfNAmd4Rddu3alX1TAnDlz6m6LLl26RD171vUULcHrr7/eKe1Fr1696nmKlqAz/IJtd2VU2nZX9aHdu3dvXH755R2/qjZg/PjxnXKePn364JprrumUc+WVY489tlPO07dvX1x77bWdcq48M2jQoDfqfY4+ffpg0qRJ9T5N7tl1113rbovevXvjN7/5Tb1Pk3v23nvvTvGLq6++ut6nyT1Dhw7tFL+4+OKL632a3LPffvvV3RY9e/bEhAkT6n2a3DNmzJi626JXr1647LLL6n2a3DNixIhOqaNoizD77LNPRbagdJwQQgghhBBCCKkhVY1oE0IIIYQQQgghtca57BmfURSVbSd5zQhHtAkhhBBCCCGEkBrSlCPayy2X/f2f1ttRaS+Ilff1119XeHXtRahM9e+hbTVZtgjlEbusqyl/C5Z/fVhWvyA21dRNle6jYR21bHTEPiF0WbPcs0krX+v9aln9wnp/on0K0BbNQ0frJctWlZY1y7+657qz/CK0T63giDYhhBBCCCGEEFJD+KFNCCGEEEIIIYTUkIZLx7VEQKQBaXJk2Tbtd0taYEnNtIRA0mmT6ttJUm6VZaisLfuVpgWrXHX5yu9pEo52kt+EZK8hW6TZRQjZopEymzxQL7/IqqPatcxDU4Ua0V6EbNFutmoWW7RTuYckrmnlu/zyyxdtl7WtYJXxl19+WbZP2rtTu9rFsoXVTqe13SG5stV2i11oi461zaH3qLT6JquOSiv/VrVFSOYdqqMsv+iIdDxkC5227FsrOKJNCCGEEEIIIYTUEH5oE0IIIYQQQgghNaRTpeOVymhE2gQAK664ok+vsMIKZXk6fdtttwEAfvKTn/i8yy67zKeHDBkCAPj3v//t87744gsAxTKor776yqdDkqi8Ykn9QraQ8geAlVZaqSxPp2V/XW66XKW8pfyBgl20LdpBKmjZQiM2SCtryy/qYQu9j9AK5R8i5BeWj4h/lP5uSQF1uVr1kdhCb1eNzD9PhKZMVFNHiT+ktRfWuSxbWO2F3i7NLkIr+chzzz3n0xdffLFPv/rqqwCAo48+2udtv/32Pv3OO+8AAA4//HCfp20R8gvxB20LyaumvWgFQhJX8QHtC9pHpG7SdZS1rS43XcZZfqHbEMsurSZhrtQW+lnX5W7ZQm8bsoXYQNvis88+K8trJ1uktRdSlh9//LHPmzhxok8/8MADAIAtt9zS56288so+/fnnnwMAJk+e7PN0uXbkPaqV3qms9jTNL6zpK1Y7ndZ2W+1FpXVUmi0sv9Asi104ok0IIYQQQgghhNSQuo9ohwI/WKOkundv1VVX9elVVlml6F8AmDZtmk9PmTIFALDZZpv5vK233tqn11prLQCFHj8A+PTTT8vypOcKKPR46J4Pff15Gk0KBRuwRoN0j55Oi1103r/+9S+fPvvsswEATz75pM+TsgaAI488EgBw1FFHle2vbWH1SNWjx6mzCQXhsHrytF9o7rnnHgDA6quv7vNmzZrl07NnzwZQXNYjRozwaeld1OUesoX0CrbayKpli9Aoqa6PxEbaL7TdrJ5Yqwdcl7ukdb1kjRzleYQiqze8mjpK28JqL6qxhZR7qL3QfmG1F5o82UXbZMaMGQCAX/7ylz7vG9/4hk/369cPAHD33Xf7vPfff9+n58yZAwAYOHCgz9tqq618ukePHmXn1yMP8uxb7XTIFq1QR1nBzNIUTlYdpN+jJH3KKaf4vEcffdSnt9hiCwDA7bff7vMsv9DtuaT1O4C2hU4Lea2vQrbQdYzUPdoWq622Wlla51lKA11Wli0++eQTnydpyz6APfLaSrZIay/efvttAMAPfvADn6dHt7t06QKguN7SdVy3bt0AFPvKfvvt59NSH2kfqPQ9Std1mjzYwmq7xRZpyj8pl2OPPdbnzZ8/36f33HNPAMBee+3l8+SbASh+/xJCbbf4hX6P0mmxgb6ftHa8WjiiTQghhBBCCCGE1BB+aBNCCCGEEEIIITWkbtJxK/BZSGYjkiYto9GSJ8l/7733fN4NN9zg0yJ5+tnPfubzNtpoI58WaYGW2VjyQY1IC9ICSeWBrIAdWoJhSTC17FLbRWxw0003+TwtmRFJji5/fS6RgYhcp/T6hND6nXm1hSXjT7OFpLXM6Zhjjin7XcsvteRpnXXWAQDce++9Pk+njzvuOADFwYss/9VYa56nrUXf7Fi2CEkBLYkyUKivQsHQ0uoTqaOsYHZp5WvZIk/lr7ECpoRsEWov9D5Llizx6R/+8IcACrJnADjnnHN8WvzKCsKS5hciC2yFNWz1PcoUrCuvvNLn6aBBYgstu9T10YQJEwAA119/vc874ogjfHrkyJEA0v3CCr7ZkbY7b3WUXK8V8C+tvRAf0G2A9pGpU6cCAF555RWft+GGG/r0TjvtBKB4KpK2hdjaCtpVTXuhyUM7nmWLkExfl6VOi410G/LHP/7Rp/v37w8AOOmkk3zeLbfc4tNrrrkmgOpsITLytDLPky1C71H6nfS6664DUCwX3nbbbX1aZMqbbrqpz9O2kKkuegqeFYArZItQ250mI28msuTiOp3Wdi9duhRA8becbi9ee+01AMXfF3379vXp0aNHA0gvN/En61sv1AZoCbo1Tbgj7QZHtAkhhBBCCCGEkBpStxHtSkdRrRHTtJ5Y6QW54IILfJ4eET399NMBFAdZ0aMJWUt1WWH6gULvhhW8Ki9YvX+V9orrstKBzX73u9+VncdSJ6QF8fjHP/4BABg+fLjPC9lCelp1Xl5HKCxb6PLTtpBy+c1vfuPz9HMvvdrab6ygT7p8tC0mTZoEANhkk0183sknn1y2nZVO84s82EKoxi+kvrJGTvV+aUvrWHWQriOtZXKkB14HC3n44YfLrnnXXXdNvcdmJBQQ0FIXhNoLbRexm64v9HKP4kNjx471edpvOtJeWAFVNNYxmw3LFjLaoEcVLL/Qo0HaVhKA6Oqrr/Z5ixYt8mkJMDR06FCfF1I4iS2sZcB0Om8B0CoNWpoWNFb8QfuFroNefvllAIXgsADQq1cvnxalR2g5HUt9YC09pfP1deTNLlm2sFRNQKGO0e21HtEWH3rjjTd83jXXXOPTEoBLj8JadZwuS7FFWjA6uea0ZfHyQKVKj2uvvdanpY6R9yUAuOOOO3y6e/fuAID777/f53Xt2tWnDzvsMADFbcjxxx/v0xKAdu211/Z5YquQX+T5PcryC7FLWh0lStcnnnjC5+ngi1LusuQaALz11ltl59bvBlYZdqTtrkewa45oE0IIIYQQQgghNYQf2oQQQgghhBBCSA2pqXRcD7lnrUdbTQAuHczgzDPPBAAsXrzY55122mk+vc022wBIDyaQdU1p673lTSYuVCrL1GWlg2099NBDAIDp06f7PC1dFfmTListbwpJXrLkJmm2KL2fvJC1RjBgB47Q8qILL7wQAPDiiy/6PC37Ex/SZa7XD7Suw5L96cAR1jVZ/p1nKpUrW/WVrqNCAZrmzZvn0wsWLAAAPP/88z5P1hgGCj62cOFCnycBQ2RfoLgOFLvpwCIHH3xw2XU0M9aUCkt+FgpMp39/5513ABQHVNHlfv755wMoDoZmTTUKBXlpBb+w2gvLFtoXrKkuuj3X22633XYAgAMOOMDn3XPPPT4twW/WX399n6dl6pW23ZYt8moToPJn0HqP0lNabr75Zp9+7rnnABTb54QTTvBpKffHH3/c54ncHAAOOeSQsmuSawlNmckbll/o+5IyDE3BS1snW+pumfYIAAMGDPDpNdZYA0Bx4Do9JUDacau9soJyld5TnghNqZC0npJy1113+bS8s55xxhk+T6T5QKGMtHRcAgMCBUm/lvlPmzbNp2W6l0y7AwrB1lrh+yI0Hcp6p03zC3mG9ZQVbRd5nrUt9HS5UaNGAShuuyu9plB7UQ84ok0IIYQQQgghhNQQfmgTQgghhBBCCCE1pG5RxwVL7mFJb3RaD+dPnDjRp998800AwJgxY3yeXgdP5LJpcgBLGmBJnFt1jVp9/yIZP/vss32elor17NkTANC7d2+fp6NlylqMgwcP9nm63EUWmCYd0VGVs6jUPnkj5Bdasj9z5kwAxbbQZS3Sbx0BU6+LKtE29Rq22hZCNWXZqn6RJQ/Uab3P22+/7dNiKy0p05L8b37zmwCK1yw//PDDfVpsrKWAMn1GVl0oPebs2bMBFKKiAs0rHQ/Js0LyQF1viC20rWbMmOHT0nYceOCBPk8iwwLAq6++CqBYfmZNVQnVQZXkV/p7M5C1Rq0lFwZsW2ikjtpjjz183jPPPOPTEn1Wr1s7fvz4smuqhtDazXnAsoWkdfmHbKGnlUh9cvTRR/s8XR9Zq7u8/vrrPj1w4EAAQJ8+fcquN+QXeYturcmaUpEm47fWf9f7y0ouekUX/U77t7/9DUAhIjZgr3Mdaoet3/NsC8Gqo/QUSD31VKbb6alGen8p16eeesrn6XdiKbcRI0b4PC1nlvPLOwBQvJKLRV7fo0JTdLIi9AP26iyWLfbZZx+f9/vf/96npZ045ZRTzGvqCPVsLziiTQghhBBCCCGE1JC6j2hbWBPUdfree+/1eXfeeadP77333gAKE+GB4h4ra7L7JZdc4tOyzqzuvZVePd27p9Ot0CtuISPauqxlFBso9MTqXiYdYGuLLbYAAIwbN87nycgaAEydOhVAcRAQjYxMWeWue2zT7NKK6LL+y1/+4tMhW6y77roAgB133NHn6edV1iTUo6AaKzBeyBbWeVqBrABpALBkyRIAwC233OLz9GiEBBKSgEEAsPnmm/u0FUTQ6tW1bJHmCzJ6LiNNrYKUhahngGJVxhFHHAEAuPHGG32eDhgngTJlzU6gOACXjNyFgmdaa9RW0160ko9YoxaAPZJtlZuuT7TqRtZL1b+HRuGsY7aaLazrzQr0o9Na/aHXo5VAm0OGDPF5+j3q1ltvBVBcr0lQLn38kF9YtrTWtc0zWaPcOq1/f//99336+9//PgDgyCOP9Hn6nUqUBLoO0+WaVUelrZMt6Va1xZ///GefZwXnk0C/APDRRx/5tLQNug3Ra25fd911AIB11lnH52k1lARL+9Of/uTz5H1h7NixPq9d36NCQY71c22NaF9xxRU+Lbb67ne/6/N69OhRdsxlbbu5jjYhhBBCCCGEENKE8EObEEIIIYQQQgipIZ0qHQ/JMkXGMXnyZJ+38cYb+7QEDdKSKEuyI3INoHgdtrlz5wIAttpqK58n8pqQ/EyTV5mNdS+6/HWQOQnyoPfRUrM999wTQLEE5Lbbbis7ftq6moIuS7FFqPzzLAW0sPxCB+CSMtT3quVL55xzDoBiqZj2C5FSpUl35LhdunTxeZYss5qAK3lHrwl/9913+7TIzrSkSdY5BwrrQ2pbWEE60gI2WhJZyy/uu+8+n37ssccAAL/4xS8y7ymv6DWAb7/9dp9+5JFHAADf+c53fN4vf/lLnxaZvm4v+vfvX3b80Fqa1prz1ciV80alcuVQuVnPcEimn5ZnlWulbXeebWFRqVxZAmkBhXXKAWCXXXYBUBxgS5eRBE7T9tVrmksQNF1Hii20r4WCyuaNZZHx6+fyoosuKjuOTIMp3dYKzqnPJdek/SrkF9b95NkupQwbNsyn9fu/vEe99NJLPu/555/3aflu2GGHHXzeqaee6tOy5rP2Oz2NVd6ZdTA0Wf88Tcaf1/co6xm1SGsvrHci/bzK9BYdgFmX9U033QQg/Z03a6qR3ic0vUWzLHbhiDYhhBBCCCGEEFJD+KFNCCGEEEIIIYTUkLpJx7OG2dNkMFOmTAFQkHgDwBlnnOHTOiq2oCW2clwdhVZHvZb99flDEtlWIEvyJOv6AsXRMB988EEA4UiB+tiyFipQiACopR9aRi52t+SFIclT3tBllCWRrEZ+tvbaa/t0t27dyo4pkiUAeOeddwAUR47VtpCozqNHj/Z5lcr4Ww2RQ+rVCnREXll3XqLuA8VlJJKntHW4pdxD60dakqcXXnjB52l5mqwXnYc6LM0XsuooWW0CANZff32fHj58OIDi9eV1uYmMVZe1SPuBwlQJbR/LFlY01Haro6zfqpGOy/G1X2hbSB22ePFin2e106EoslnPUR7Jup+09yhJa7m4lkvK6hR6f1nZAijUM7q90LYSQn7RCmvKh7BsodPy+7x583zepEmTfFpk5LLGM1BsC5E26xVFQn5hTdNoNb+w6ly5Hz01VMvA33zzzbJ9t9xyS5+WdkRHstbvx6G1n2V9bj2lQqZv5KFtriWV+kXa9AVJ67IeMGCAT8uUgFVXXdU8fpZfhGxRD7/giDYhhBBCCCGEEFJDajqirXsPrPUDpadV533wwQc+LevP7bHHHj5PBx2yejn0aISsLTxjxgyfp0e0JYBXaE1BjRV4JA9YIxRW79u1117r8/Tos5Sb7im3ym3WrFk+T/fKyvG1fXRPkhw3FJBDX3NebWFh+YXu9ZZRaL2tLh8JSAPY68cvXLjQp2W0SPuNtqUcS9uvVf1CI/em19KcMGECAOBb3/qWz9PBzmSUJ61cpIxDI9pWQBsge5RuvfXW83njx4/3aUvx0Aq95lJGG2ywgc/beuutfVrX7RZSLvq5Hzx4sE9LYDUdTM1aQzPUXrRqHaURW4TWzk577qw6SttSAnPpYFp6PVupm6p5xlvNBoLYoBq1mS6LbbfdFkDxaJClFNDr0mpVidUe6bbdwvKLkKKlGbDeT6y2O01dIPz617/2aT06ffDBBwMovC8BxSPaciyt5NH1WdbawM1aph0l9E4rZaUVfpdddplPy0izLj9LDaXb62rabvm+0AHYQgG28tReVKp6Auz2IqSA0ljKP61olsB0oRHtaoKd1dMGHNEmhBBCCCGEEEJqCD+0CSGEEEIIIYSQGtKp62hbcgIt8xbpd9oabSIN0NIanX766acBACuvvHLZMYFCYISsQApZ6dL7SDtWs6KvW8pok0028XlaOi6EJLKDBg3yeVpqJuWmy2fNNdf06f333x9AsX0sQnKOPMjPLKxn7N133/V5OjCdJVeWAGZ6f71u6tSpU31aAtno/XXADpHk6HW0ly5dmnqdrYLcjw4sJuX63//93z5Py/os+aAEQAMK5apl+FbAxpDs0pJpaZmUPr+2Zd4JTTWy/D3tdyl33UZsvvnmPn3NNdcAsKe06OOHrrNVfcSyhSWLrWR/ee61L+ggg9L2aPmfXue50vO0gozfeu4s2WXIFh9//LG5/4Ybbgig+D3p8ccf92mpz3S9p9sGa43akDQ8r7bQZMmV02SxMoVLt8d33nmnT0udrm2h6yt5ZxK5f+nxs6aypF1TXmX8FqH2Qr/TSpuc5jdS92hbVNN277TTTgDsb4Jzzz3Xp3/0ox+l3k+eqcYvQjJu+b7QvqDrIJkeUI0cvfTaStPVHqcaOKJNCCGEEEIIIYTUEH5oE0IIIYQQQgghNaRu0nFLvmIN0w8cONCnL730UgDAvffeW7hAFXXu+OOPB1AcEfiTTz7xaVmfUK8fOHLkSJ8WSYglu+2IBCEvZEme0iRDIn9JKxdZ71Qi96ahbWGt69gRyVMrYNlC+4KWir388stl+99xxx0+PW3aNADFUktL7qwj+mq581FHHVW0nb6mjkpF8yQ/e+SRR3xaVjzQMm2N+IW+1yuvvNKn58yZAwDYZZddfN6wYcN8WmSboWj+Fq30/JdSaXsRKquQfM2KKDt37lyf169fv7LjV1NHtRJpa2YLVnsR2l9Lw/W0ITmWXkdbr7yw7rrrlp0zJIFtBbLabo0uF5Hk33333WXHAeyVEe67776yY+opKbq9sPwiNM2jlbAijFtybqAQgVpHt9brPFvR+K3pK9oWU6ZM8WmJRq5XZmgHvxBC92hF09dYqyhU0wbpY+qpkaXHfPHFFzOvU5PX96hQHWW13Wn2s/xCv2dZ01csu1TTXjDqOCGEEEIIIYQQkhPqHgwt1GOg10YbN24cAODyyy/3eXrkToKd6dGiV155xaelJ1ePnI4aNcqnQ73urYR1X5Yt0kYlrB4h3UslgVZ0r/nQoUN9+sknnwRQPIqato5z6TWl2SRPvXshQs+dfoalN1vfvw7YIdvqfXS5S77O06NJVk9vq/qFRu7xgw8+8HlPPPEEAOB73/uez9PlY5XViSee6NNSrnqUVKs+pL7afvvtfd7o0aN9Wo+4ll5nOxC6V13+8lynrSds9Xrr/YcMGQIAmDhxos/TQWuyRrRb2SZZo/ZpgYBCo0Wyn95O10dW2/2HP/zBp4899tii7YBi1Y5FlmqqNL+ZsNppa5QtzRYSSFMCcQHF6zBLcEytBhQlDgAsWbIEQPEoqigKgI6pbkLBj/JAR/xCAmS99tprPm/ffff16WOOOQYAsM022/g8ec8FCrY8+uijfZ4eER8xYgSA4va8HeooS0kQ8ovQWu9We6HrIzlX2jGtNc2FDz/80KfFvwC7vc8blaq8rLZb20q33VKGeh9dH0k7YL3n6v1Dbbf+PVQ3Wb5eKRzRJoQQQgghhBBCagg/tAkhhBBCCCGEkBpSU+l4pWs9psmVDznkEADAN7/5TZ93ySWX+PSbb74JAFi0aJHPExkUUJAma1mmlthqmUHWNVUjJ8gDWVKiNFmlJcOxJDkHHnigz9tyyy19WgJMaTmHlthmlWva89FKtrB8Rd+rBCgDgPHjxwMoLsuQVFMH8xK/0MfXMpyOlGsr2EL46U9/6tMyfeWqq67yeWPHjvXprl27AiguSy15EjmrlvcNGDDAp6UO01MuTjrpJJ8+9dRTAQBrr7122XW2UpmXkhV8L60+kPo8LSCKBNN66qmnfJ4O6iRTBp599lmfpyWap59+OoBi2WzomlrBRlntRZos07KF9gvZVttCBz0V6ave/4UXXvBp8Uu9NnSfPn0AAOeff77PaydbaHTbIGsH632ee+45n54/fz4AYPr06T7vpZde8mk9xUvQAWgrbbtb4T0qJOO32m7tF7169QJQCIoGABdffLFPn3XWWQCK13vW76ziF3rtZS09X5Z32rzZpCPrsofqqKxpEIAtPQ7V/dZ16kBp2tZa8p9XKm0vqmm7hffee8+ntfx+nXXWAWDLzUvPm3VNneUXHNEmhBBCCCGEEEJqCD+0CSGEEEIIIYSQGlK3qOOVrqtoSW4GDRrk83SEcTmmlo5L1EagICMXyV8pWdKPNAlDXmU2mkqjAobWudO/i5xSS2DPPPNMn15ppZXK9hG5h6YjtsgbofXLLVv079/fp6+//noAwC233OLztGTmjTfeAFCIUAoUy//knGkyqZtuuglAQZ6pt9XSq1bzCykXPVXllFNOAQBMnjzZ5z366KM+vffeewMA9tlnH5+n1zCV8kpbi1MkmLqsdVTyX/3qVwCAn//85z5PbN1ucuWOrMWpbSVTAvQ0Cj3VSFZJ0L6kfUhkn9b508o/JEVsVkKyzJAsUNIzZ870eX//+999+uWXXwZQHOnamt4Suj5dvv/85z+L/gWKfbEV/KIj61NL26uncs2YMcOnJYL7vHnzfJ72C2HMmDE+3bdvX5/WMnWhVdtujTW9JfQeJfe98cYb+7xJkyb5tDz3OoL+q6++6tP77bcfgOL3gUoluqE1hvNMpe+0Vrmk+ZI812nHtNph65oOPfRQnydrPy9dutTnabl4nt6j0trmWn1f6LQ8o1JXAcUrTgwfPhxAcRuij591/o7WS8tiI45oE0IIIYQQQgghNaSmI9qVrt1s9RLpdNq6qJLWParSewsUAgh169bN5+neu6x1VUMjd3kjVO5W76yVTvv9o48+AlDcO6dtIesDDhs2zOfpAE9WT97ixYsBFAc90IEjrLXx8kooyIouSxmZ0L3aelspN937p9dnlF5zXW7ahyQolOUXaT3hrWADQdti5MiRAIBRo0b5vPvuu8+nJUCTDqCmy2XgwIEAigPa6NEKsYVW5ehARDLKutpqq/m8Tz/9tOw8raAuCI3ShUa5pT3Qo6GDBw/26SuuuAJAsS10kDrJv+eee3zeb3/7W5/u0qULgEL5AwUbWGt2avJmC01WcBtd/roMZORGB9jSbbeUtRW8VKd1uekRWVEq7Lbbbj5Pgp5uvvnmPk+Pegh5C8pljQyFVFGWX2y99dY+74knnvDpuXPnAii2hQR5BArPs6gQALsMrTWE09YYzmvbHbKFpPWzHqqj9LaSr/d57LHHfLpnz54AipUaFlbQr7TgqXltLzRZttB5CxYs8GlRdWilzcSJE31abJH2/VHpOsx6zXSxQZotWmGt80ptEfIL63f5zgAK7TFQeL9Ne4dotjqKI9qEEEIIIYQQQkgN4Yc2IYQQQgghhBBSQ+oWDE2w1qGzJAJAQS4bktncdtttPk9LnrRMWbCkr3otVZHZ6PUILVlgnmU2gnUP2haWlC9N2iHrOurgNlrmLchatgBw1113+bQEhXr77bd9nsgDdbCQSy+91Kc322yz1PvIG9XZMPCvAAAZ9ElEQVTYIiSz6dGjB4DiKRNaOi5+p597/YzPmjULQLHkSdZ9TJM8tZJc1rpuLUOSgDQAcNBBBwEoLkvtAxJMS0v/58yZ49PyDOty1UG3RM6p5cohKWAr1FFZ157mF/KM62ddtxc777xzWZ5OCyNGjPBpXd+IX2jZZju0F1n+kCbVkzpI/gWKy+3b3/42gGJZrKwpDxQk/aNHj/Z5Wiq4//77Ayiu4z7//HMAtrQfaA1bCKH2wnqP0oFitd8sXLgQQKEuA4qDAF511VUAgOeff97n6XZc2gZdB4ottF9YEs12sIWuj8QWuj0Ite3altIe6N91uUs6ZAur7c+bLULBzgRtC5k+BAD/93//B6B4ysSzzz7r0/JOqoOjaltIO//xxx/7PAkkCxSC1eqpYDK1UqYAAMW2tNaTzptdhGreaUNtt2y7xhpr+Dwd+OyAAw4AkB6sWZ5xywe0L1A6TgghhBBCCCGE5JCajmhbgQFCAZaswGjWyCpQCPo0e/Zsn6cD3Vjn1D0a0vOtA6ZIuppejjws4WJdb6iH2bKF7mXSvU+y/NDPfvYzc38p96efftrn3X///T4tttDH33333QEUL++2/vrrl91T3oLbaKxn1Fp2QBOyheynA2hpxBbS0w0U98q+++67AID58+f7PDmW3qdVR+6sOirkF1pJIyM8QKFXXNtq22239Wk5ly5LXcbSa66DDEo6VEflmazgJSFbpLUXlt9YS4joc+6xxx4+fc455wAoDpgjtkgbuWslvwjdlx5hOO644wAAJ5xwgs+zlAT9+vXzeTrwnLQHegRD+4iMCGn1iPhNNaqbvJEVlDL0HqWX5Prxj3/s01bQJ72coNCnTx+f1iNyS5YsARB+j7LaizyP3Mn1Ws9bmi2krHV7bQUq1bZYb731fNp6d9PthfiN9guxhW5DdH2VV1uEAvJZ9yXvlADw1FNPAShWXk6YMMGnZaRbB8zUI7JvvfVW0b9AcbA1CSor/gEAvXv3BgCce+65Pq/V3qOyloIN+UVa2y1prWrSPiLlruul0LdepW13PeCINiGEEEIIIYQQUkP4oU0IIYQQQgghhNSQugVDy5JIWsEidL7+XSOyJC0H0NLxnXbaqez3pUuX+rTIa6qRn+VZ0iFYkicpSytAgc7Xv2tEZpNWbrL+nQ6y8t577/m0yGl33HFHnyeB0U477TSfp+3bClJAyy/EFrr8tbwlZAtBH1PvL897mi1k3XKZmgFU7hd59g9L8iTlpiWQ2i6ybcgWabI8KU9L/qfTOk+uJbSOdt6wprdYfpFmC0mH7j+trKwpFXr99IsuughAcVAokeOmraNtXUse7BOyhZRRWntRqV9o2aaWyFoBG3U7bkkBrTqqFdaXt+SW1bxHVRp4TJ9Hy2ElWNp2223n87QcVt6p9LuVXFOadDxvNhCsAEtWMDIt09bvLFYwNI01fWnTTTf1aWmndXutt5UpYNo+4itp077yaguN1XZb77R6KpDUHRK0DLDl6HpqqvVOK+9LQPE7lXD++ef7tEyD1OXfCtJxq46ypjd09FtP0N8H06dP92mZolfNt15IOm75Yq3giDYhhBBCCCGEEFJD+KFNCCGEEEIIIYTUkJpKx60hdy2TEKz18ABb6mVFFV+0aJHPk3VpgUKEOi2BtWSZlqQmTS5eTzlBPQlFHRcZRTW2sKSrWuaky2jcuHEAgAEDBvg8bReJaCrrowLAzTffDKDYPjqybV6lgNb1WlJAbQvrGQzZQqJeAgWZE1Aod+03+ndZx1kiZAK2XDnP0d4trCkVIaxomjqto5QKlqRKSw2tyJiWBLPVprRosqLrhvxCl/8qq6zi0yFbSFnr8teyMvGHn/zkJz5PImWH/CLP9rEi+srzmNZeWBJILRUUW+io4rrcJXqsrsO0XbOiWqf5Yl5tYNXz+r6tNc2t/bX9tF9I9F5dB+npEbK++VprreXzrHcq3U6L36TJ+PMqkQ3ZQstVLaz2Qr/T6EjKQvfu3X1a1i9/9NFHfZ5+5xJpbDVTKlrJFtoH9PNoMXLkSADA0KFDfZ5+7idPngygWKavy1VkzjqSuW5jvv3tbwMAunXr5vPEL6zVNPJG2uofkq/bzkqPldZ2y7N7xx13+DxdB0laTxPoyLdeWttRep3LCke0CSGEEEIIIYSQGlL3YGgaq7df97pJ75EVTAIo9P79/ve/93m6d0J6onSPo+5lsXqXQmsY520k28Lq4bZ6lHS5SU+QFcAAKNjie9/7ns8766yzfFp6WqWXDwCmTJni09IrqM+57777Fl1bVjqvZJV72vrvMvKg/cJaL/j000/3eVopIMfSIxg6ENHYsWMBFK9ZaAXUaYXyt7DKPa2OskZBtS1Co6hWQB0rOEcrBEwJYfWQhwLuhfzCqqPSlCBW4Dt9/AsuuAAAcMABB/g8Cdio1xhuVbt0xC/0qIK2heUXhx56aNnx9f5WcMfQmvetZgvBCmZlPcuA/R6l02ILCXoGAI8//rhPDxw4sOhfoFgBJddiBXiylHSl+XnFUkBZvmIFutTlp9+prOCB+vjSTusAXVp5Jn5HW2TXW4DdXuiAfzvssEPZeSxVT1p7IddirVmeNoqdV1ukPVuCVV9X860nddRee+3l83SQOVF6SFBmfU6dtpQEoWuvh004ok0IIYQQQgghhNQQfmgTQgghhBBCCCE1pG7ScSFLVlD6u+RrWaaWnFnBPywZQJo0wJKJWzKBVpCLW1RTViLz0ME6dPlbtjj55JN9WtZ11Hknnnhi2bWEAjzlVVoTIhQgzZIFaum3ZYsePXr4PL3moCWpqtQvWrX807AkaZZfaHmgJYu1jqnTaXWQ5ZftYAO5R/1cW+s5W7ZY1vYizRZdu3YFUBz8RqRu7WCfUB1l2ULXUSFbWOXeEb9o1fIH7HuzZKhWe5HmFxYPPfSQT1t+oe1qSTCta2pVu4QCpIXWPNe2CE01GjFiBADg1ltv9Xk777xz2ba0ReG+reklgF1Hhd5pQ+9E1ntUaPpTq5H1fhn6vgi13fL8A8Dw4cPLjqXXyQ6dPxTYup424og2IYQQQgghhBBSQ/ihTQghhBBCCCGE1JC6S8c1MmSvJQJWBPC09SFDUsCQ9LidZOIhQmVhSbuX9fjV2KqdCN23FeXVkjylHSd0fNqinNCa5SHJWdqxWNbZWJHIQ3LietlC0no1hax9WxlLAllLW1h59JVs0qZd1dsWWe9M7Waf0BQ4KSs9XTJkC82QIUMAAIMHD/Z5aVMvrfO3E6FpFpJOm9JS6XtUNdLjdrJFpauIAB2ro6xzLev3XWfZhyPahBBCCCGEEEJIDenUEW2hXUeRm5FaBvXRPVLt1JNXK9ohwFLeaYfgJs1CR0YIqukVr/ScxIb1VWOoNEBaiI6MchObUDCyauB71LJBFWvjCdVRy9pOL8t1NAKOaBNCCCGEEEIIITWEH9qEEEIIIYQQQkgNcdUMrTvn3gfwRv0upyX4VhRFPet9EtqiImiL5qLu9qAtKoa2aB5oi+aBtmgeaIvmgbZoHmiL5qEiW1T1oU0IIYQQQgghhJBsKB0nhBBCCCGEEEJqCD+0CSGEEEIIIYSQGsIPbUIIIYQQQgghpIZU/aHtnNvdOTfDOfcv59yHzrk/OOd6V7H/KOfcAufcatWeuxE451Z0zj3nnIucc2NLfjvZOfesc67TOyyccw8l12T9/bXCYzS9LZxzxznn7nbOzXPOfeKcm+2cO905t1LJdg2zRXL+tvAL59zI5D4/dM4tcs496pzbv2Sb/3TOzXfOrd6ga9wtxS8WV3GMXNgDAJxzRzrnZjrnPk1sMsM511/93sh6al3n3OXOub8n1xc55/pVeYztk33Xqc9VLjuV1seN9A3n3EHOuT85595I6qmXnHO/cM51qeIYTe8Xzrm5Gbb4rdquoW2Gxjn31+T6zqtin6a3BeCfu1nOuc+SZ/+K0meu0bZol/ZbcM37Xts2bXez+wXb7tq13VUZzzn3HwDuA7AYwCgAJwIYCuBB59zKFey/AoAJAC6MoujTai+2QZwGYK2U334LoBeA73be5XhOALBLyd8pyW93hHbOkS1+CmA+4mdtJIAbAZwLYHLJdg2zRbv4hXNuOOJnaz6AwwEcBmABgNucc99Rm96ebHN6p19kMT9EsX/sWclOebEHADjnJiB+9u8C8B0ARwB4EIB+yWhkPbURgEMALAIwvYPHuBDAtVEUzavZVdWeSuvjRvrGaQC+AnAWgOEArgIwDsD9lbzI5cgv/hPltrgw+U3bopF+4XHOjQGwdZX75MIWyb3dDOAZAPsDOAfAGAC3lmzK9rtzadb3WqGl2+48+AXYdgO1arujKKr4D8ADAOYAWEHl7QggAnBCBfuPAvA5gDWrOW+j/gBsAOATxB8VEYCxxjYXAHi+0deaXMs1lZZvXmwBoKeR99PEHhs0gy3axS8ATAHwFoDlVd7yAN4GMLVk2xMAfABglQZc525J2e/Zwf3zYo9dAHwN4IAKtm2Ubyyn0mMTu/SrYv/tkn22bHR5d+Dezfq4Ub6RUpcemZTvsAr2z4VfpFz7gwDe1XVXkt/Q9htAd8Qvb2MSO5xX4X65sEXSLj5UkndQcq8jmsEW7dJ+q+tt2vfaNmq78+AXbLtr1HZXK0cYCOD+KIq+lIwoimYmJ/7PCvYfC+CvURR9KBnOub845x5Q/3fOufedc59r6YdzbrJz7gn1/0Odc9OSbZcmEoyyXh/n3InOuRcSSdAi59yTzrlKrhWIe/z/CODRjG3+CGAL59ygCo9ZF5xzqwI4GMCdunwzyIUtoih638iemfxbKkdplC3axS9WAvBJFEVfqfv8CsBSlKtjbkL8EnlgBfffbOTFHuMAvB5F0e0V3FNDfCOKoq+X8RDHAXg2iqLnJcPFErs5eiPn3FOJ1GsjlXe+c+4955xL/r+3i6ehvOtiOdts59ypzrnlS451WGKnpc65j1wssfx+NRcdqI8b4htV1qUWefGL0mOsB2B3AJN13ZXQ6PZbXqKnVrlf09vCObcWgA0B3FPyk8gxS/dl+8332mWl6W2RF79g2127trvaD+2vAPzbyP8cwFZZO7pYgrMbyiUI0wAMcgWJzgAAPRD3hAxR2+0O4G/q/xsAuAVxr9wBAO4E8Dvn3PHqnIcD+BWAqQBGJNveAmDNrGtV++4A4IzApk8D+BixFK+RHAigC4DrQhvmzRYGuyIeyXu5JL9RtmgXv5gEYCPn3I+dc2s553o6534KoB+AK/SGURQtBPACGusXk51zXznnPnDOTUleuDPJmT2GAHjGOfcjF8cw+DJpgA42tm2WeqpahsO2xYZiT+fcGgC2AfAvAMPUdsMA/C1KuqIR2+JBAMcgltlfh1iyd77s4JwbAuAGAA8jttfBAK5G3LhWQ2p93CS+Ieya/PtC1kY584tS/guAg902NswvkmftSMSjJNXslxdbSKdGadv4RXJNpW0j22++1wqt3HbnxS+WFbbdasdqhtOfAPB4Sd63EH/0fB7Yd2fED9FeJfnbJvm7Jv8/CcCzAO4H8Iskb7Nkm+Epx14OwAqIC/UZlX8FgH90QDawBuK5p2OT//dDisQm+X06gPvqLWcIXPO9yTWvUMG2ubGFcfwBiJ3y6maxRbv4RbLvCMRzdqLk72OUSJ3Utn8A8HJn2kKV3UUA9kX8IXESgPcAzAPQq1XsAeCzpPxfRzxffi/E874iAPsb2ze0nkKV8jMAvZPtjyvJXzPxre8m/z8geSavQTKFAcDqiF9cjk85tkts8eNk3+WS/NMAfFiDe82sjxvlGyXXsE7iF/dXsG1u/MI4/otZx2mEXwBYEcDzUFJxVCgdz5MtkufrxpK8ock1vNQktmiL9hs5eK9F+7TdTe8XJedn2134veq2u9oR7UsB7OScO88518s5t1ly0q+TvyzWTv4tla89A+BDFHozhiHu9ZhWkvcFVO+Ic25j59xU59y85LcvED8Mm6pjzwSwjYsj5+3pKo9CeCGAVxEbvxLeR+H+Oh3n3NqIg0VMjpT8KYM82cLjnOsL4M+IbXNKymaNsEVb+IVzbiDiHsO7AeyDuFfvLgA3O+d2N3ZpiF9EUTQriqLToii6M4qih6Mo+jXia+2NOMhKFrmxB+LGvwuAUVEUTYmi6H4AoxG/wJ9lbN/QeqoDmLaIYjnXsygu94cRz7WU53Ao4sZ4muznnOvrnJvonHsD8WjCFwDOQ9zj3SvZbCaANZxzN7g4wn61veGV1seNbjNWR1yXfgng6Ap2yZNfeJI6a1MA/y9js0bY4gwAq0KNyFRBnmxxKYCDnHM/cM6t6ZzbHrF0+SvYbSPb7zZ+r22jtjsPfrEssO3WdOBr/1zEI4oR4gdiKuLIbK8F9pNgH5sbv90K4BHEgZUWI47CNxDxS0BXxKM0j6rtVwcwF8A/EcvCBiGWw1wT31JRz8f3EfdYfoV4BOhWZPTKIO4V+xJxb1r35G9Acu3jk/+7kn0mA3i1Hj1JFdrkR8n1bV3h9rmwRcl19QAwOznXOhnbNcQWre4XyX4zAUw38mcAmGXknw/gy0b5hXE9/wRwbwvZ410AHxj5l8IYiWmUb6jzV9srvkuy/T7GbxcDeDNJP4c4UnCfZPstEL9UzlPbL5c8v/MQzx37j8QW55VeE+KgNA+j8HL1AIABVdxnsD5upG8AWAXxS8yHAPpXuE9u/KLk2q5E/GJWFgiuUX4BYD3EbcXhKLxjdE/K98IkvXzG/rmxBeK4HhOT80eJP/0awJMApjXaFuq8Ld1+I4fvtSXX0mptdy78Qp2fbXdhm6rb7o4W+jcA9AfQO/n/CwCuD+yzd3IDg4zfxiOeD7Nr8rB2TxziI8QSkvdRLLHaKznWkJLjXKedoOS3NRCP9ryNEplQyXZHoSCLTfvrXrLPPVnH7AQnmA3g6Sq2z4Ut1PZdEVdA7wLYKLBtw2zRyn6RbPsZgIuN/EsAfGbkXwVgQSNskXL9LyAOlNIq9ngAwEIj/7IUezS6nqq2sd4k2f4w47d9k98k8nr/JP+fAH4A4CkAN6jtN062P6LkOD9PuybEL1sjESsE3oGKwhq47mB93CjfQCxZvgtxAMOBVeyXG79Q+6yMOKDV7YHtOtUvUIisnPW3TYvZQj7s1kyewY8A/E+jbVFy7pZtv5HD99qSa2mptlvt0/R+kZyfbXdhm6rb7g4tgh5F0SdRFD0XRdECF6+tuxni9d6yeDH5dwPjt78h7uE5G/F8h8VRHB10OuLejrWgZAQorBH7hWQkk+r3z7jmRVEU3Yg4alxWgIu/IpYw6L8xyW8XJf9fWrLP+gBeyjhm3XDO7QBgS1QQBE2RF1sgkeLchbiM946iaE7W9migLVrcL4B4GZodjfydEPc2ltIwW5SS+MkmAB4PbJone9wGoEdyb3L85RBLn2Ya2zeNPSpkLuLOHcsWjyB+YToXwELEDSQQl/+BiAOshGyxIuJRRZMoipZGUfQXxCMPfRGrajKpoj7udFskz8ZkAHsgnsP/WBW758kvhH0Rv8A2my2eRvk7hsgmb0jSWe1c7myRXMezUSwdPRZxJ8i1xqZsv/leW0SLtt2yT9P7RQeZC7bdRRdUTa/Gtojn/g1P/s5D3Ev0vxXuPxfAZSm/LUDcO3GByjs1yfsMwKoqvyfinp8nEUegOwSx7n8OimUdkxBHBDwIse5/LOKeq9uqvO9+SAkagbhH6mvrt874Qzx69QWSXtgq9suFLRD35H2NuEdyYMlfz5JtG2KLdvELdd4piIOijUC89EQE4Icl2zrEo0kVrQtbY3tMTmxwIOI5QKcirtDfBLBWC9ljFcS9wG8glrftg/jj+ysAezSDbyTnPij5uyopp3HJ/3etYN+HANyR8tsTyfFuUnmjUBihWV/lr5TYdU5y7v2TY8+B6hUH8D+IG+dDE1sclmxTNjUi5ZqC9XGjfEOV/3kor0vXbRW/UPvfgdjvV8rYpqHtd8m1RJU+E3mxBeKRwZOTf/cFcDni+qlsfepG2QJt0n4b19YPTfZei/Zpu5veL5Jzs+0u3qZDbXe1hb4l4vmYixHPZ/kHgKOr2P9/kTLnBcCNSaENV3kSLfAhY/thAGYl1/Eq4kAJ55Q4wXcTg7yHuOJ8HbHMtWuV990P6RXS4YmT9miAE6yYOPWdHdg3F7ZQjmf9HdUMtmgnv0jK+HHE0R4XJekxxnaDk2vcqgF+cSbiRvEjxBXnW4gbxL4taI++iEfBPkye/b8jVn5YdmtUPZXmv2XlZew7DvFIyzdS7BRBRSdFIarpXGP7bRI//RSxvO9/UCKJQ/xSdS/iaSqfJ8/ONQDWruBaK6qPG+UbiF9W0mxxTov5RU/Evn95YLuG+YVxLREq/9DOhS0QS3VnAlgC4BPEazfv20y2QBu13yXn6ocme69Fm7TdefCL5Nxsu4u361Db7ZKdOwXn3IaIh9x3i6JoRqeduI445+5BPE/yvxp9LdVAWzQPLWqLqxBXRv/R6Guplha1R159oyvihvWEKIpuaPT11IK8+gb9onmgLZoH2qJ5oC2aB7bdar/O/NAGAOfc1Yh7p0Z26onrgHNuGwCPIS740NzhpoO2aB5azBZ9ALyGuOf4kUZfT0doMXvk3Td+jDjIzNZRZzdYNSbvvkG/aB5oi+aBtmgeaIvmgW13TIeCoS0jZwOY2ZE1MZuQPoglRrlzgATaonloJVv0A3BqHj8kFK1kj7z7xsUAbkEsk887/ZBv36BfNA+0RfNAWzQPtEXzwLYbDRjRJoQQQgghhBBCWplGjGgTQgghhBBCCCEtCz+0CSGEEEIIIYSQGsIPbUIIIYQQQgghpIbwQ5sQQgghhBBCCKkh/NAmhBBCCCGEEEJqyP8HUl1OOSObT78AAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"print('Misclassified Digits:')\n",
"fig, ax = plt.subplots(nrows=1, ncols=10, \n",
" sharex=True, sharey=True, \n",
" figsize=(14, 8))\n",
"ax = ax.flatten()\n",
"mscls = [y_test_preds != y_test] # indices of misclassified digits\n",
"for i in range(10):\n",
" img = X_test_standardized[mscls][i].reshape(28, 28)\n",
" ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
" ax[i].set_xlabel('%s (was %s)' % \\\n",
" (y_test_preds[mscls][i], y_test[mscls][i]))\n",
"ax[0].set_xticks([])\n",
"ax[0].set_yticks([])\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It's easy to see how even a human might have difficulty identifying some of these examples, hence it's not surprising they were misclassified."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***\n",
" \n",
"## 6. Testing on My Own Handwriting!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next I photographed my own handwriting to see how well our CNN's performance generalizes to an unseen writing style. The digit images were manually rescaled, centered, and cropped prior to being loaded below:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA9sAAABvCAYAAAD8OtMiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAG8ZJREFUeJzt3Xu0VOWZ5/Hfo0YFEVAOeNfDpYVIq5jWCcZMMG23iZ2MlwlpozEmcbVrXOPEmJXOOOnYK2066Cwn3Qa1Y5teHbpNa0Zj2rQa7NGAQqTFC94QvESUmwKKoAZBBH3nj13ve546pw7ntqt2Ve3vZy0Wjy/n1Hk8u/aueut99vNaCEEAAAAAACA/uxWdAAAAAAAA7YbJNgAAAAAAOWOyDQAAAABAzphsAwAAAACQMybbAAAAAADkjMk2AAAAAAA5Y7INAAAAAEDOmGwDAAAAAJAzJtsAAAAAAORsj4F8cUdHR+js7KxTKu1j5cqV2rhxo9XzZ3As+m/JkiUbQwhj6/X4HIv+41g0D45F8+BYNA+ORfPgWDQPjkXz4Fg0j/4eiwFNtjs7O/XYY48NPquSOP744+v+MzgW/Wdmq+r5+ByL/uNYNA+ORfPgWDQPjkXz4Fg0D45F8+BYNI/+HgvKyAEAAAAAyBmTbQAAAAAAcsZkGwAAAACAnDHZBgAAAAAgZ0y2AQAAAADIGZNtAAAAAAByNqCtvwAAAACgnU2bNi3Fr7zySs2v+fCHPyxJWrhwYUNyQmtiZRsAAAAAgJwx2QYAAAAAIGeUkWNQOjs7U/zBBx9IklavXl1QNgCAZjVx4sQUr1ixosBMgOZ04IEHSpJCCH1+7d577y1JWrVqVV1zKpMRI0akeNu2bT3+3cxS7I/RokWLJEmf//zn09jPf/7zeqSIFsbKNgAAAAAAOWNlG4Pim0X055NY5GP+/PmSpKeeeiqNfeMb3ygqnbbwuc99LsX33HNPirdv396v7/efeH/oQx9K8ZgxYyRJa9euHWqK6EOta5A/LsjPyJEjJUnvvPNOzX+PlU7ePvvsU9ecMDj77ruvpOqVPH8uzZw5M8W33npr4xJrYv714oEHHkjx7373O0nS+++/P6jHHcz1avfdd09xXO2Wej83kRk+fLik6tf43XbrWnscO3asJGn9+vV9PtakSZMkdV0XUdsZZ5yRYt9MbuvWrZKqf//+uRzPC3+sdu7cWfNnxOoEf47++Mc/HkrauWFlGwAAAACAnDHZBgAAAAAgZ21VRj5hwoQUb9iwIcWx/KBWeZvUVaYwbNiwNPaRj3wkxeyf1xMlmsU4//zzJUnr1q1LY5SRD83SpUtrjsfysFhSJlWXOsVywRdeeCGNvf322ymOx8iX+u2xR9clt6OjI8W97eEJFOWoo45K8W9/+9se/+6bno0bNy7FjzzyiKTqUj9fphxvtfANifx54c+X/pRxom++jN8fi3g98+99jj/++BT/4Ac/aEB2reXee+9N8ZYtW1I8atQoSdLUqVPTWK33Sb5MP36PJM2dO3fAufj3vP41JJ5DU6ZMSWPLli0b8OO3kz333LPH2JFHHpniZ599dpffH0vPpepjWKuZGnqaN29eiv2tFgcccICkgTVYvuSSS1J84403pnjTpk2SpJ/+9KdpjDJyAAAAAADaFJNtAAAAAABy1lRl5LNmzUrx7bffLqm6q+KOHTtSHEvDfdmmL03wcSzl8eVptfhud7609LTTTpNU3akYKEIsG/MlY9OmTUvxk08+2fCcWp0vA6+HyZMnp3jNmjUp9re6xJJaX5aInnyJ65IlS1I8ffp0SdJDDz2U2886++yzU+y7Dvvj1o7Gjx8vqXoPX1+GHLsuD8SJJ56Y4liu6R/Hl2X60ttar9l+LHbT3muvvdLYq6++OuD82lUsnfW/38MPPzzFK1eubHRKLW8wz/96eemll2qOx/P1+eefb2Q6Tcdft/w54OcS/eWvMX7ecdNNN0nqusUPtfnf+UUXXZTi2bNn9+v7/ftc/57NH+N461Jv3cqLxMo2AAAAAAA5a+jKtt9v9txzz5VU/cmcX8WOq8z+E4pan373tqeh/xTqoIMOkiStWLGi5tcec8wxkqTnnnsujflPL3/zm9/U/L4yY2/tYsRmfX4/51rNi9A8eltdOOmkk1Kc54psO/P7y/uGN3PmzMn9Z/mV840bN+b++M3kox/9aIpjoxrfGHCoq/mDfX6feuqpkqpXYH1Vz1tvvSWp+vXIr3zH8d72cG3HSpK42i91vT8a7N7PaE2f/exnJXVViErVFVbtvuK93377Serf3sz9tXnz5hT768ndd98tiZXtvvjrcn9Xs6Wu561voOav4bEpmtT1vviQQw4ZdJ71wso2AAAAAAA5Y7INAAAAAEDOGlpGftZZZ6V4+fLlkqR333235tfG8q/eypXjvpDHHXdcGnv44YcHldfTTz/dY8yXoW/dunVQj9vOfBmzL9VBY+y///4p9iWusQHXYYcd1vCc0LcxY8ak2DdZ8dcb9M6Xww6myQ1qe/zxx1Mcn4vN0AjO72ncXzNmzEhxfL74PXTb8RYo3/TM3443f/78ItJBwWo9x997770CMilGPAf8+9Sh8ns3e301XkbGl/F3dHSkuNYtWqNHj05xnF/4Rmj+e/w1Lj7v/eM3C1a2AQAAAADIGZNtAAAAAABy1tAyct/te9u2bZKkkSNHprEPPvggxbEMxHf98x3mYoe6G264Ibf8zjvvvNweq91NnTo1xY899liBmZSTL/H0ZUzHHnuspOoOjSjG8OHDJVXfZuH3EI7/LlWXlKOneO33v78f/vCHKb700ktz/5n+tacd+ddeX3ba6p3xFyxYUHQKDec7tI8YMSLFJ598cgHZoGiPPvpoj7GXX365gEyKcfrpp0uS7rjjjjTmS8onTJggaWBd2b/+9a+n2L82XHbZZYPOs0xih3hJevPNN1PsdxWJ9tija2o6btw4SdKqVatqPu68efN6jDXjra3t/W4CAAAAAIAC1H1l2zdy8s3QYhMWv7rz+uuvpzh+2hE/1ZB6/2QjL/fdd1+K/Sp7u69wDEZvje3QeH7PQb8/PPJ31VVXpfjKK6+U1FWlI1WvEMZPZydNmpTG2n1/03qJrw2+kVw9VrM9v4rejvw1vFZ1DJpfXKHz151Pf/rTRaWDJvHaa69JKu97V7+/eOT3n1+xYoWk6lVVXzlbqwpgy5YtKfbfN23atKElWxK+GtPvSb506VJJ1a/tJ5xwQoqvu+66XT5uqzzHWyNLAAAAAABaCJNtAAAAAAByVrcy8mOOOUZSdemFL3WKpWp+f0+/h2os+ah36bjnb9r3ufpSd6DZHHjggSleuXKlJOk73/lOGps1a1ajU2orDz74YIr97zVeI/z14cgjj0zxE0880YDsyiHe1jNs2LCG/Uz/etSOezO34/9T2axZs0ZSdfOn2267rah00CTiLSJjx44tOJPm4W+xu+KKKyRJV199dRpbvXp1iuMtYIcddlga87eW+lv3MHA33XRT0Sk0HCvbAAAAAADkjMk2AAAAAAA5q1sZ+YsvviipuvTCd/BbvHixpOoOdIceemiKlyxZIql6vzT/tXmKne982aDvROs74wHNJnbWlLo6M8bzB0M3ffr0FPu9Infs2CGpuhv5008/neJYiua7ZfrdGdavX59/sm2uER3CDz74YEldHX3b1Sc+8YkUL1y4MMWxM7l/3vq9mzdv3tyA7NAf8T1LR0dHwZmgaL7bdjx3eY2p7bvf/W7V392NHj1akrR27do05ucH/noI9Acr2wAAAAAA5KxuK9uxQYPfvzOuGHh+5Xrnzp1diVVWhfzK+FD5n/W1r30txcuXL+/xtb4Bwrx583LLoV3keVyQn3i++aZeGJp4LZKkN954Y5df6xuo3XDDDZKq9zPeuHFjzccdNWpUvx6/rOLzOs995M8888wU/+pXv0pxbBzmVy/acf/63l7XZsyYIam6ean//4+rZn7l2zfo+tjHPrbLx8fQxOuK1PVc/eUvf1lUOijQF7/4xRRv3bo1xRMnTiwinbbhmyVHvqrKv44D/cHKNgAAAAAAOWOyDQAAAABAzupWRh7Lm3zpRV97q9UqI8+zIc43v/nNFN9xxx0pjmWefg/Xyy+/PLef2478fsKxDN83uPONJZCfyZMnS5LeeuutNObPkVjezx66xfB7mve1v3lnZ2eK4/niy3GnTJmS4qVLl+aUYWs64ogjJFU3A/QNNy+++GJJ0p133pnG3n777RTH88KXQ/tbYXzJeCwhPO6449KYb3zX7hYsWLDLfz/llFMkVe8jv2XLlhTff//9kqqPTxyTpJNOOimXPMvqsssuS3Es5T/kkEOKSgcF8rcP+Ns6XnjhhSLSaTu+oam/nnnDhw+XVF3GD3THyjYAAAAAADljsg0AAAAAQM7qVkYeS1t9V+/YpbQ3/mu7P04e7r777hRv2rSpx8899thj05gv1UJPvgw/li/5UkIMjS/Tf/nll1Nc6/YML5bG+pIm3/Xa7w4Q9+X0HU1nz549lLQxACtXruwx5vdKffbZZ1Mcy6hXrVpV97yaUSyL/MlPfpLG/I4S11133S6/P54ve+21VxqbNm1aimt17/dl6OjSV5fxmTNnSpLmzp2bxvye3rHssh07vDeCv55H/vn7hS98IbefdcABB6S41i1KI0eOTPFLL72U289F//idLvyxQj7879eX6X/rW99Kcbxd7PDDD09jq1evbkB2aCWsbAMAAAAAkDMm2wAAAAAA5KzuZeS+VMx3+962bVu9fnQPHR0dkqo7OPvOghMnTpQkLVq0qGE5tSNfZoOh6a2bu+/YX0s8x/yx8KV+sdOyj6+//vo05ncM2Lx58wAyRh789XLSpEkpjrcSjB8/vsdYmVxwwQU141rd94d6C9I+++xTczyWRC9cuHBIj9+qDj74YEnVt335EuLbb7+9x/f4kv1nnnlGUnXn/R07duSeZ7vyt8DF5/hASscPOuigFG/cuFGS9P777/d4zO5xrXPMv0bEjv7cTlZ/8XYYf0vBq6++WlQ6bcs///05csUVV6T4xhtvlMTvH7vG7AgAAAAAgJzVfWXb86vZ8VNS/2lRrcYfA3Huueem2Ddnic2i/OOfeeaZKf7Zz342pJ+LjN+3FkPjV3r8HsB9icfAN0Jbt27dLr/nq1/9aop7W81D4/nVwNhMjaZdjeP31vbnUxn3sPUNGzds2CBJOvDAA/v9/U8++WSPMb+yHZumSdJdd92V4rinN2qLFUz+uv3OO+/0+Dr//PWv07E64TOf+UzNx/f7ONfiKwTfe++9fmSMwfJ7Psf3zX1VuqH+YsNI32C5s7MzxbUaoaJ8WNkGAAAAACBnTLYBAAAAAMhZ3crIY9mSLxP3Ypm5L+0+6qijUrx8+fJdPv4tt9wiqbpRgS+X9SXrsVzN7z2M/MRmHeybmh9fnjeY522tZja9mTNnzoAfH/Xh90p94403Ujxq1KgeY+gy1GZoA3n8WmW67W7FihUpjs/FV155ZUiP6W+V8Q0dv/zlL6e4t0aRyEyZMkWS9Nxzz6Uxv5f89u3bJVW/z/Klx7H0+7777ktjvT2/L7zwQkm9v15cfPHFA8odfYu/c6m6wW9s6oviTZ06VVL1awS3e6E7VrYBAAAAAMgZk20AAAAAAHJWtzLyuLevL3vsq/TFl4x973vfk1S9f+f69etTHPfqfO2119KYL1nfd999U+z3Fkb+YgngQEqXsWujR49OsX+OT58+XZK0ePHimt8XO836MnQ0p7Fjx6Y4lgj6TsHjxo1LMXt4FuvQQw9NcRlLBH2JpL9Fqx6P78tlsWvLli3rMebLyOPtfL7bu+8iv2rVKknSV77ylTTW0dGR4nfffTfF8bj7x+LWsfryO+X4Wy3KuCNCEfx7Wv/79y699NIeXzts2LD6JoakVeYdrGwDAAAAAJCzuq1sz58/X5J02mmnpbFaqzN+JcfvFXnNNddIqm7WUWtPQd/4wzcXmjFjxmDSxiDET5bq3aSoTHzzoTFjxqT40UcflVT9XPf7m8ZjsXnz5nqn2PbGjx8vqfq65Stm4vPdf7IaGxL5uK8mkVLXfrdbtmwZatqog7LvlXr22Wen+LbbbpNUvXezj2NVm39++3Mkrpb6VVMvnncYHH8N2m+//SRVv4/yFYT+uNXiV/Piaw5VNvV11llnpdhXkfhKJzRGvJZJvTcn3bhxo6Tqa9xQm0ei/2qtbMem2M2ElW0AAAAAAHLGZBsAAAAAgJzVrYz86KOPlsQ+mWUQG6qsWbMmjd1zzz0p9rcSYOB8+VIs5du0aVPNr+3s7GxESqUwYsQISdW3qvTVbNGXXcZSJt8Y0pfIzp07N5c8gXq7+eabe8STJ09OY/7aX+scqdVoyDfaKmPTuUbo63aia6+9VpK0aNGiNHbrrbfWNSfs2hNPPJFif96sW7euiHRK7ZxzzklxPFckaf/9909xvG2DprTF+NKXvpTiq6++WlJX48dmwso2AAAAAAA5Y7INAAAAAEDO6lZGjvJYvXp10SmUxoYNG4pOoTSWLl1adApA03r++eeLTgFDdMkll1T9jeL5XXn66haP+oq7IknSnDlzUuxve4n72vvO8WicKVOm9BjzO/Q0C1a2AQAAAADIGSvbAAAAQMGWLVtWdAqooa/mqChebEobqw2aCSvbAAAAAADkjMk2AAAAAAA5o4wcAAAAANCStm7dWnQKvWJlGwAAAACAnDHZBgAAAAAgZ0y2AQAAAADIGZNtAAAAAAByxmQbAAAAAICcMdkGAAAAACBnTLYBAAAAAMiZhRD6/8Vmr0taVb902sYRIYSx9fwBHIsBqevx4FgMCMeieXAsmgfHonlwLJoHx6J5cCyaB8eiefTrWAxosg0AAAAAAPpGGTkAAAAAADljsg0AAAAAQM5acrJtZoeZ2e1m9paZvW1m/2pmhxedVxmZ2aFmdp2ZPWRmW80smFln0XmVjZnNNLNfmNkqM9tmZs+b2VVmtm/RuZWNmX3KzOab2Xoz225ma83sNjM7qujcIJnZv1euU98vOpeyMbOTK7/77n/eLDq3sjKzPzGzhWa2pfJ+6jEz+8Oi8yoTM3ugl/MimNm/F51f2ZjZSWZ2r5m9VjknHjezC4rOq4zM7JNm9mDlfe0mM/upmR1QdF4DtUfRCQyUmQ2XNF/SdklflhQkfV/S/WZ2TAjhnSLzK6FJkv5U0hJJv5F0arHplNafS1ot6S8krZV0nKS/kvRJM/tYCOGDAnMrm/2VnQ8/kvS6pMMl/S9Ji83s6BACjUcKYmbnSDq26DygSyQ96v57Z1GJlJmZ/TdJ11f+/LWyBZhpkoYXmVcJ/XdJI7uNnSjpbyXd2fh0ysvMjpH0a0mLJV0oaaukmZL+0cz2CiHcUGR+ZWJm/1nSvZL+n6TPSRqjbL43z8z+IISwvcj8BqLlGqSZ2deVXYAmhxBerIyNl/RbSf8zhPC3ReZXNma2W5zImdmfSfoHSeNDCCsLTaxkzGxsCOH1bmPnS/pnSaeEEOYXkxkkycwmS3pO0p+HEP6m6HzKyMxGKzsG35B0i6RZIYTLi82qXMzsZEn3S/rjEMKvC06n1CoVaM9K+nYI4YfFZoPuzOwfJZ0n6aAQwqai8ykLM7tS2eLF/iGELW58saQQQjixsORKxsx+LalT0pQQws7K2AmSHpF0cQjhRwWmNyCtWEZ+uqTFcaItSSGElyUtknRGYVmVFCumzaH7RLsirhwd0shcUNMblb93FJpFuV0taVkI4WdFJwI0gQskfSDp74tOBNXMbJikz0u6i4l2w+2p7HV6W7fxN9Wac6ZWNl3SfXGiLUkhhEeVvZ86q7CsBqEVnzhTJT1TY3yZJO6JBLrMqPz9bKFZlJSZ7W5me5rZ70m6UdJ6Sf+34LRKycw+Lul8ZeWaKN7NZva+mb1hZrfQc6UQH1dW6fEFM1thZjvN7EUzu7joxKD/KmlfZZVpaKx/qvx9rZkdbGajzexCSadIuqa4tErpfUnv1RjfLun3G5zLkLTcPdvK7ofcXGN8k6T9GpwL0JTM7BBJ35P06xDCY0XnU1IPS/qDSvyipD8MIbxWYD6lZGYfUvZhxw9CCM8XnU/JvSXpbyQtkPS2st4SfyHpITM7jvOjoQ6u/Pk/yo7BCmWrqdeb2R4hhNlFJldy50t6TdI9RSdSNiGEZyq3u9yhrg9nd0i6KITAh+WN9byy1e3EzI6QdJBarEqwFSfbUtYUrTtreBZAEzKzEZL+TVnToa8WnE6ZfUlZ05sJyu4Bu8/MPk4/g4a7TNIwSbOKTqTsQghPSHrCDS0ws4XK7sG7RBL30DfObspWT78SQvjXytj8yr3c3zaza0OrNfVpA2Z2sKQ/kjTbl8+iMSqVaL9QVi17kbJy8jMk/b2ZvRtCuLnI/EpmtqR/qewccq2yxdYfK7v9paVuYW3FMvLNyn7h3e2n2iveQGmY2d7KupdOkPSpEMLaglMqrRDCsyGEhyv3CJ8iaYSyruRokEp58nck/aWkvSolgaMr/xz/e/fiMkQI4XFJL0g6oehcSib2kbiv2/i9kg5QtnqExjtP2XtzSsiLcaWyVdPPhhDuDiHMCyFcIuk2SbPNrBXnTS2p8sHG9yV9U9IGScslvSJprqR1BaY2YK34pFmm7L7t7o5SdiCAUqqUy/5C0n+S9CchhKUFp4SKEMKbykrJJxWdS8lMkLS3pH9R9mFs/CNl1QabJR1dTGpwTLUr1lA/y3oZj1WCLbVy1EbOl/RUCOGpohMpqaOV/f67lyk/omzrqXGNT6m8Qgh/KalD0jHKOvOfI+n3JD1YaGID1IqT7TslTTezCXGgUvZ0ktiPECVV+bT1ZmUrqGeEEBYXnBIcMztA0hRl90WicZ6U9Mkaf6RsAv5JZR+CoCBmdrykI5X1OEDj3FH5+1Pdxj8laW0IYX2D8ym9yrkwVaxqF2m9pGlmtme38Y9KeldZfyg0UAjhnRDC0hDCBjP7tLL3Ui21i0Ir3rP9D5L+h6R/M7PLlX0a/teS1ihrgoMGM7OZlTA2gzrNzF6X9HoIYUFBaZXN3ylrbjNL0jtm5ptKrKWcvHHM7A5Jj0t6WlkTqCOV7e28U1lzKDRIpaLgge7jZiZJq0IIPf4N9WNmN0t6Wdn58aayBmnfVlYaeF2BqZXRXGV7nt9oZh2SXpI0U9KpotdHUc5X9jpxS9GJlNj1kn4u6S4z+5Gye7ZPl3SOpGtCCLW6Y6MOzOw4Sacpe72Qsh0UviXp6hDCfxSW2CBYK/a/qNyHd42kP1ZW8jRP0qU0HiqGmfX2JFoQQji5kbmUlZmtlHREL/98RQjhrxqXTbmZ2WWS/lTSRGV7dq5RNuG7imtUc6hcs2aFEGjI1UBm9m1lb1qPkDRc2SrSPZK+G0JoqXvw2oGZjZR0lbJJ9n7KtgL73yEEJnsNVrkN7FVJi0MI/6XofMrMzE5T1lhzqrLbkFYoa8x1Ywjh/SJzKxMzm6psEfX3Je2lbBvb60IIcwpNbBBacrINAAAAAEAza8V7tgEAAAAAaGpMtgEAAAAAyBmTbQAAAAAAcsZkGwAAAACAnDHZBgAAAAAgZ0y2AQAAAADIGZNtAAAAAAByxmQbAAAAAICcMdkGAAAAACBn/x/euRR3TrYI5QAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# specify the path of our custom digits\n",
"imagepath = './myDigits_v2/'\n",
"\n",
"# we need to invert the pixels to harmonize \n",
"# format with our training data \n",
"def invert_png(pixel):\n",
" return np.absolute(pixel - 65535)\n",
"\n",
"invert_png = np.vectorize(invert_png) \n",
"\n",
"# load the image files and assign class labels\n",
"my_digits_X = []\n",
"my_digits_y = []\n",
"for file in os.listdir(imagepath):\n",
" img = Image.open(os.path.join(imagepath, file))\n",
" pix = np.array(img)\n",
" pix_inv = invert_png(pix) \n",
" my_digits_X.append(pix_inv.reshape(784))\n",
" my_digits_y.append(int(file[0])) \n",
" # note: first character in filename identifies class\n",
"my_digits_X = np.array(my_digits_X)\n",
"my_digits_y = np.array(my_digits_y)\n",
"\n",
"# sort the images in ascending order\n",
"sorted_ind = np.argsort(my_digits_y)\n",
"my_digits_X = my_digits_X[sorted_ind]\n",
"my_digits_y = my_digits_y[sorted_ind]\n",
"\n",
"# display the imported digits\n",
"fig, ax = plt.subplots(nrows=1, ncols=10, \n",
" sharex=True, sharey=True, \n",
" figsize=(14, 8))\n",
"ax = ax.flatten()\n",
"for i in range(10):\n",
" img = my_digits_X[i].reshape(28, 28)\n",
" ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
" ax[i].set_xlabel('%s' % my_digits_y[i])\n",
"ax[0].set_xticks([])\n",
"ax[0].set_yticks([])\n",
"plt.tight_layout()\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next I standardize these digits, obtain digit predictions from our CNN, and display the classification results:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test Accuracy: 100.00%\n",
"\n",
"Classification Results: (predicted class is shown below image)\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA9sAAABvCAYAAAD8OtMiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJztnXm4XEW19t/lAIpIQExCAgQNES6BMDwgg4IMYoCgxiuR6QE+pguBQBg1IINKIIGAgUCABBKIIuBE0FyVK9MlfOgNKNwvTAFNSAhhCmNUVBSs748+q867T9emzznpPru79/t7njwUdbr3rq61V1XtqrdWWQgBQgghhBBCCCGEqB/vK7oAQgghhBBCCCFEu6GXbSGEEEIIIYQQos7oZVsIIYQQQgghhKgzetkWQgghhBBCCCHqjF62hRBCCCGEEEKIOqOXbSGEEEIIIYQQos7oZVsIIYQQQgghhKgzetkWQgghhBBCCCHqjF62hRBCCCGEEEKIOvOBnny4X79+YcCAAY0qS9uwcuVKrFq1yhp5j7XXXjusv/76jbxF27B8+fJXQwj9G3X9ddZZJ/Tv37DLtxXPPPOMbNEk9IUt1F90jyVLlsgvmoRG+0W/fv3CwIEDG3X5tuKPf/xjw22hNqp7LF68WLZoEhptC/Xd3ae7fXePXrYHDBiAK6+8svelKgnjx49v+D3WX399nHXWWQ2/Tztw4oknPtvI6/fv3x9Tpkxp5C3ahjFjxsgWTUKjbTFgwADZopsccMABDfeLSZMmNfIWbcPBBx/cUFsMHDhQ46hust9++zW8jZItuseoUaNkiyahL2yhvrt7dLfvloxcCCGEEEIIIYSoM3rZFkIIIYQQQggh6oxetoUQQgghhBBCiDqjl20hhBBCCCGEEKLO6GVbCCGEEEIIIYSoM3rZFkIIIYQQQggh6kyPjv5qRvr16xfT73tfeu7g3XffBQD86U9/6pMylZWPfexjMf3+978/+Zl33nkHAPDGG2/0SZnKynrrrRfTeX7xr3/9C4Bs0WjYFmaW/Izb4s033+yTMpUV2aJ5WHfddWM6r7/wvlu2aCzrrLNOTNfqLzSOaiw8pq3VRskWjeXoo4+O6by69vOor7nmmj4pU1lp9b5bK9tCCCGEEEIIIUSd0cu2EEIIIYQQQghRZ1pKRj58+PCYdhnBW2+9VfN7IYTMfwHgz3/+c51LVy7OPffcqry5c+fGNNc1Sz5StmhGyUcrsdVWW8W0SwC783zLFvWHbeHPfU/aGrbFqlWr6lewEiJbNA9XXHFFTN9www0AeiaBlS3qxxZbbBHTGkcVC49px4wZAwB45ZVXan7vAx+oDN3nzJkT82SL1eOMM86I6X/84x8AgLfffjvmrbnmmsnveXs0YcKEmHfJJZc0ooiloR37bq1sCyGEEEIIIYQQdabhK9uDBg2KaZ4ZWnvttQEAH/nIR2IeB+bwv/OqKAdR8ZmLFStWxDz+rAdW4fw11lgj5vnMIFCeAFGf/OQnY/pDH/pQTHugmo9+9KMxj23hgc+4fufNmxfTPgvIgT0YtoVfl5+F559/Pqa7M6vbDmy44YYxzc+lB6pZa621Yh7b4tlnnwUAvPzyyzGP7eoBItgvmFq2YB977bXXuvNTWp7LLrsspv/whz/EtPsFB/7jdmOTTTapyrvjjjti2m148sknxzz2IQ8WCKRtwdctiy3YLz74wQ/GdC2/8LaL81L9xXPPPRfzZIv3Js8Wxx57LICsLbh+NtpoIwBZW4wYMSKm3W+WL1+evG+tNqqMtthggw1iup7jKG+b7rnnnuR9d9hhh5i+8MILq+5fxnHUxRdfHNNLly6Naa9rDuTE9TNkyBAA2frnvn/bbbcFADzwwAPJ+/o4iznooINieuXKlTHN47N2ht8vuC69P8jzi9NOOw1AdgV0t912i+kPf/jDALKr3YyPswDgiCOOAAB84hOfiHn8DJTFLwYPHhzTqTEtr/b7OBYA+vfvDyDrK3fffXdMu43uv//+mMf1z37h9t5mm21i3jHHHBPTr7/+erd/T73RyrYQQgghhBBCCFFn9LIthBBCCCGEEELUmYbLyFlOwfIZlwGwZIylBw7Ln/72t7/FNAcucFgOyvKyO++8E0D2XE/+u9+DZQwskZs+fXrVvVoRl2sAaYklSyldEgV01g/bgmXKKSkfS2r4uk8++SSArGSdy+VnFvJzwZLpdpHk5PmF24J/P0v+v/e97wHIBovYfffdY9p9hOVRLO9h+c3ixYsBZG3NWwE8zWXhYBPtEkyN68qfPwBYf/31AXTKYoFs/bgN+ZnkZ9W3ROyzzz4xb6eddopplmOecsopALK24LNvPc32a0db5J29XMsvUuducn+R8guWIHK9LlmyBEBWglhGW5x11lkxzdt7vD13OTiQbWP++te/Vl3rhRdeiOl9990XQLaett9++5hmG5900kkAsv1FyhZ5QR7bxRarO47yegSytvS2i9ui0aNHx7RvGQCAv/zlLwCyftGT/qJdzoTm+uM2yOudt3Wx3bw/YTkzj2O//vWvA8j27QMHDoxpbm9cvj527NiYx+Osww8/HEC2jZsxY0ZMt4st8voLryuuE5YTf/zjHweQ3Z5x/vnnx7RLk9kWHBSYpcter+wXHHCwLH5Ra0z72GOPxTzemuf1M3Xq1JjHfuF9N9vi17/+dUzPnz8/pl1+zt8//fTTq+7FtuD6b2R/oZVtIYQQQgghhBCizuhlWwghhBBCCCGEqDMNk5G7vOOll16KeSz1cvnSlltuGfNmzZoV0wsXLgSQlex4NEcAeOqppwBk5d4suWF23nnnzD2BrMwhxbJly2LaZQgsc2glXDLDv4lt4XXMEleW37h8IyXLBDqlsyyj+ec//5ksy6abblr1/ZQElGGZekq620q4fCbPL/y3cvRetoX/7gULFsQ8j6wJAGeffTaArF+wZIZxGSjbohYuJQQ6fbxVpZr+LHF0WbaFR65kW/BWE0+zZMmlgkCnLV0eCGT94u9//3tMn3feeQCAmTNndrv8LKtyeVSznCnZU1J+wXhdshSSI4t79NGJEyfGPLaF11V3/ML7GbcJADz44IMxfeutt1Z9h23R6n5xwgknAMi2sRyN/KabbgKQ9Qv+u/et7BeXX355TLsElm3B9+L+4JxzzgGQrUuW/XnUYL7/5MmTY7rV/cLLn9dfeL3wec3cX+y9994AsnW24447xrTbhW3B/Tjjz3UqKnYe7Be+FaBVz4N2W/g58V3xZ5htwc+l9xfc7rOc1vsbtgVLx5mNN94YADB37tyYx9s3Ro0aBSBrq9TpDa1ui1p+wdHEue/2dw1uo7iu3V/ybME+5ieY7LrrrjGv1pi2nfzCx1F5tvDn+tVXX415u+yyS0z7+JX781Tf7X1B1+uzj336058GkG3veGteCh7TNrK/0Mq2EEIIIYQQQghRZ+q6ss3BMnwWjWcgfv/738f0zTffDCB7lievvPpqK888pWZOPMhTVzjQka/YTps2Lebxat64ceMy9wSywV98loRnIVOr9M0ElzVlCy7zZpttBiAbbCIVNIBn8/icPIeD4HCag9u4+oBXKjiYhK9UcJAXXw3ncvFsVmqVvpngwBEpW/Asmq+qcZ3xzJuvXO+5554xj1e5HT5zk4N28SyfKx54RpX9wm3BM8Lsr/49/ntqlb6Z4Gc8ZQt+7t0WXGf8rPrvz1sh9ety4BWGZ2pTAVl4pSK1gseB21K2YJrdFr4SUMsWXCe80uBBndhXUqsL7Bec5tU894uHH3445j3zzDMx7fbO8wsvQ556qhltwStA3jdwnXhgRqDzt3IblfIL7i84SI2vZOSt5LC/XX311QCARx55JOZxe+mrhBzg6/jjj6/6LdzfsF1+8IMfJMtQJBwALtVG8TPubQDbYq+99oppb/vvvffemJdaueZ+M88vfEyR11+4DbiNSvlFXn/RjEqQ1bUF+4V/lvuLVBvVHVv4yl+eLVx1wm0YB/jyoL95/UUzjml7YoujjjoKQOfYFuh85wA625Ce2MLHXgDwxS9+Mab9Hny2c0/8wp+RVvKLnoyjXIXBv/nEE0+Maf99ebbwwHaLFi2KeVyX9913X0yPHz++6lr8XNQa09Zqo1bnnG6tbAshhBBCCCGEEHVGL9tCCCGEEEIIIUSdqauMPHW2GstRLr300pj2QEEuMQCykhuXz3CAApf38Wf5POfrrrsuplnK45IDljawHPGCCy7IfA4AjjjiiKrPspyg2UnJIFgSwzJz/yzXGdeV2zAlB+frsnSDg4CwJNy3AuSdS+tyWv4OS/r9Hq1ki5RfpIK+8WdZMsYymNRZp/z9F198EUCnXAbIBuXienW75UmXvYzsF6lzP1vVFg5Lg1LndubJyN0WfNYqf9alZOwXX/jCF2KabeTpvDbKy8h+zWeEtrpfOLVs0b9//5g3Z86cmPb+gn8/28Lrkm1Ryy84oAv3Y88//zyAfFu0ol/wth+v69mzZ8c8rkuXDeadK+t+wdJu/qz3DbwNgAM5sS1cbpjnF25DbqMuu+yymHZbPvnkk2gVUn7BfSTLaf2zEyZMiHnc955yyikAss8i9+Nelyz5z/ML7w/ybOHXaCe/6Ikt/HfzmDW1vYLb/VR/0RNbcN/N9/Ln3YN3AdkxX7v0F3m28DocPHhw8lq9sYUHhgSyPrZ8+fLMNYGsLWr5RbvYIq/v9t/N728HHnhgTPtWIR5HeZBOoLPv5fq77bbbYprP3H766acBZG3J7VVqHMVj2kbaQivbQgghhBBCCCFEndHLthBCCCGEEEIIUWdWW0bOElaGZRQOR1YcNGgQgKx0YKuttopplxyxNIFlCH52Np9pyNdimYPLAVnWxvhnJ02aFPOWLFkS0y4taMYIjQxLLJnUmXFcP6nopHwtl1SwHIMjIXu9p86r63ovlyuy1IpxeQfLUDhKvUcpbMao4wyfE8ikIvDy+Zepekld6+c//3lMH3fccTF91VVXAeiMyghk7cu28Dqs5RcsteITAdwWzRhdmWH5HMPPs1PLFqlrsU1ZGutRl3/xi1/EPJa68fU52nMKLxd/J3Um9epEy+wLeusXHpGUt5TMnz8/pj/3uc8ByLZBLBl3u7FcOs8vWD6ewj/b6rbgrVIMnwvvcP2k/ILHAd5f5PmFt+0ux+96Tba724IltKly8fc5srq3UbVsWjTc3zGpcVSqjeI+nsdRhx56aNV12C/8vnl+kbJFXn/R3TaqGaMrMz0Z03L9cD+Zupb7BV+H/cLHTOwXebao1Xe7tJm3DPBZ995ONrsteusXvl2LZeAsTfbtDy5hBmrb4pZbbolp3rbipx3l0S5+0dtxlPsF+4e//wGd/RD//s033zymPcr72LFjYx4/97xVYsWKFQCAYcOGJcuasgWf1uM0ou/WyrYQQgghhBBCCFFn6hogjTeVe8CTMWPGVOUBnTN2vILMK9O+csqztLz647Mgzz33XMzj2V2+l89u5c2O++z3o48+mvwtPEvTzPDZdClb8Go1f9ZXB3gWj1emfTWOZ0l5xvWhhx4CkD0TnVcSOFiBlyVvZdvLlXdGbSuSskXeCp//bp5tS9U71x8HeFi4cCGA7IwuzzxywJW886Edt0XeWZytEtAjdX4m0FkXPGOb8gu2BSsK5s2bByB7piavXPssK8/Oc2APnun1a+S1Ud4GtZNfMO4XebbwlRgPgAIA1157bUz7qgS3Szz77e0RP//sF6n+Iq/db5c2itsI/q3bbrstgNptFCvVuI1PtVFsC1fC5NmC87vbd+fZqtXbKH8uU2fSA8CRRx4JILsy/aUvfSmm3RbsF1yXvoLTE7/IW02t1V+0OrVs4c8gq+6473Yf6QtbuKqEx3GtSG/94pJLLqn6DisAfax67LHHxrxNN900pi+66CIAWVuwj3E/v8MOO3TrN7S6X6zuOOpb3/pWzJsxY0ZMe11yG86K5YMPPhhA9l2PVVO8Is592nv9hiL67tZ4ixRCCCGEEEIIIVoIvWwLIYQQQgghhBB1pmG6BpdnsLSYJV1Dhw4FkJUus0TQpbHTp0+PeSzXdFiKmXcvJ08G4fl8ff5+nlSnmeHfmhfYwPHfypImlmz47+ezBTkolt+LJVOpABbdIWWjPHl8K5IXfMVJ2cLPzgY6t1qkfAHolNPOmjUr5rF8qid4vbd6nefhtshrF/x3P/zwwzHvwgsvjGlve1hqtscee8T0zJkzAWRlhbXaqN7S6jaqZQvf8sDSZg4yk2qjUttm8vqLFBwwpye0oi24zHlBibp+lmWR3EalbDFgwICq69Sz725XXGKZ9zvdB7jvZWmsSyy5fjnQrNMTW+RRq79oRb9gXJJdq7/g4Fm1/CIV1Ja3X/Bne4LLafm5SJW1VanlF6lxFAeIu+GGGwBkz85mCfLIkSMBZMexXGc9kYTXaqNa3RbeX9T6newXhx12WEynxrTsF75loCd9dx5F2kIr20IIIYQQQgghRJ3Ry7YQQgghhBBCCFFnVltGzhGNWVrhy/Es++Nocx4t8IADDoh5LF+bOHEigKwEk2V9G264YdX9/ew8ICszSElpOJqgn/PG0VVZbrDxxhtXfb8ZYTkrSzY22GCDqs+yXMIj/fJv5np1KRTXP0v+XZbGUkGOFsjyEH8e+P4sa/N7sEydP1sr2mCzkOcXKRk5/z6XLbEtuN7dH/g73/nOd2Laz9dm+RM//xzR1PP5WuwX/htSZyDz35sdLidHoUzJZVN+4dGZgayUyaPHcp3eeeedMe1+w1JCfn5/9KMfxXQtW/gzkGeL3kqe+5re2sLlsKnnE8j2E06t/oJ9hPsLj/DMUU7Zbz2qcKvbgs8sX7ZsWUwfcsghALL9Ncsxf/aznwHItiv8mz3yO9uP024L/g7XJfdjHOXc4fumzvRmWqW/4HJyf5HaAsZ16XXN/sPPeC1bDB48uOo7eeMo9z3+Pke7bvf+olbf7fVWq+/O+37KL7iN4n4mZYuvfvWrVWW99dZbq+7Z9R7NTG/9opYt/JQkPi2J25ozzzwTQPYMZ/Yx3lraXb9InUfNf292eusXPo7i+u2NX/D9/ZpdP+tl4b6rJ2PaRvqFVraFEEIIIYQQQog6s9or23mzNR7YgVdYhwwZEtM+y+Gz5EA24I3PTLzxxhsxj2cofGaCZ2E5sASvlnpZeJZw2rRpMe3n4fIsGM/k82ebGa4rxmd7eDaKZ+a83rh+fNUO6Fxp4Otz4A6f/ebZIl59GDRoUEy7vXiWlmeh8macnLzf2GzwqhjjZ2zmrRqlgkXwLJ7P9PL1+fs+S7h48eKYxzOKqSATeas/fN8Ueb+x2ahlizy/SNmCVwp8lpSv/9Of/jSmn3jiCQDZNpJtceKJJ8a0z87y9VPnTDOpld9mp7e28ICZPAvOdZKyBfuFf5bbuMmTJ8f00qVLY9r7LFbq1ArI0oq24N/PXHbZZQA6+0Ug+yweeuihALJ16SuoQKcSjK+fsgXbj/uLVBvFq91cllqBOFetWvWef28W8vo9by+4v/jJT34S0z7OOemkk2Jeqr/g67Nf9WYclXc+e602qlbf3izkjWlTtuDnOhUAjX9zaoWTv++f5Weaz+TmoFF+j3POOafq/kBtpVDeb2w2euIXPbFFyi/4+9ddd13V3/fff/+YdkUI0Nl2cRvVjn7RqDFtd/tu/g6Po4YNGxbTrnjg1XbuT1L9RV/13VrZFkIIIYQQQggh6oxetoUQQgghhBBCiDrTsHO2Xe7LssBvfOMbMe0yDt6QzpIZl8FstNFGMY9lCo7LC4FsgDOWDrjsbN68eTGPJXIvv/wygKzk/Stf+UrOL2s93BZ5Z4e7zJsDELBkyT+72Wabxbyddtopph966CEA2fOcXYYDZGVpDstwWKL57LPPVpU170zpVsRlLHlnNrpsj23BwTaOP/54AFkZPn/f/WnzzTevuifQKelhWCrI1+VtHU5vz/1sRlKSop7Ywv2CAyhuvfXWVdfkdo3rdMKECTHtMmYPcAdkpYDjxo0DkH1uenvWZDNSyxYuC2MZ+YEHHhjTo0ePBgA8+eSTMY+fVbch22LRokUxzW3br371KwDApEmTYh5fNyULbCdbXHzxxQCyzxr3jZdeeikA4LHHHot5/Kx6G8IS13PPPTemvW/hwJ3sF6k2KnXuatfvOe3UX6SetalTp8a0BxzlcRC3US7H5DaKx1Fu456Mo7i/YOms24KfG7ZVq5OyRaq/4DEt+4XbotaYlv3Cx6ZA2i9YDs42njJlStVnZYtqW/D7ScoWp59+ekzzVgseE4waNQoA8Nvf/jbmlckvavXdnmZb1BpHpc7DzvMLvq7bcNNNN415vP3Yv1fEOEor20IIIYQQQgghRJ3Ry7YQQgghhBBCCFFnGiYjd1h+9tnPfjamfTk/L1KyyztY2sHSgtT5myxX5oiZLh1kGQ6nXULFeX72druSkm5zBD+W17gMhG0xY8aMmN53332rrsnRZVlS45IOllKxbM2jCbJMpFZU7FaHn1X/3cccc0zMS0XkZ79iv3C5JcuU+Cx79jE/v/CMM86IeTvvvHNMuw05om+t6L+tCEuKUvIvlh6zX3ia6z/VRvH1ud2aOXNmTHvbc+2118a8Rx99NKaPOuooAMCsWbNiXpls4ZLuX/7ylzHv+9//fkzffffdALJ+wXgbz3/fZZddYvrKK6+MafeR4cOHxzw+ncFt2O5+wbAtTjvtNABZv+C+waOZP/744zGPJfkuF5wzZ07MY79I9d0sIWRbeBvFUWRZ7tku5PmFbz9hCevXvva1mE6Noxjupx0eR/HZzS5Z53EcR/E/+eSTAZTXFt5f8LOc6i9q2YKvX2tMO2LEiJj3/PPPV11Ltlg9W/A1WYY+cuTImL7vvvsAANdff33MO+GEE6quVVZb+Jg2bxzlNkhJx4H0OIqvxe8d3kZtu+22yc8W2XdrZVsIIYQQQgghhKgzetkWQgghhBBCCCHqTMNk5C4J4Iiwu+22W0y7RJKlARxZ0aXNDEuPXQ7A0m8+HJ6lBS5r44PSOVrennvuCaBTBgU09nDzomCZBte7Sy8479VXX41pjsTrcLRrx6OaA1l5DkeHHTp0KICsJIpxG/P925E8W7jkiKXjLJvkKO/OkCFDYvrMM88EkI0uy3XN9eoy8jvuuCPm3XjjjTHtfsN+2Y7kyZdSsj6uC5ZQOmwLb6NYxsRbIriNcltMnDgx5h199NEx7Vs8eEvA7Nmzk+VuZWrZ4qCDDop5+++/f0yn2otNNtkkpr1dYVkmb1XhbRduC45yylx++eUAgGOPPTbnV7Q2/rzntVG+xYX7UN5W5M8oP6sXXHBBTPtzz5H3J0+eXPV3oNNuHP2X8XauTG3UvffeG9MeCXnMmDExr9Y4auzYsTHtUXt9DAAACxcujGn2iy233BJA1tc4fdJJJwEALrroopq/p5XJa6N8ixf7CstVly1bVvUdHtN6G8dbvfL6C69jjrR89dVXV32P79+ONNoWAwYMiHkvvvhiTB955JEx7XWdOhmBr1EmW6TGtI0aR/E75hZbbAEgP8J4kf2FVraFEEIIIYQQQog60/AAacz9999flcezeKnZBg7axRvZfRWbV6v5bMHf/OY3Me2rrLwK/uUvfzmmDznkEADZABPtDs8yef3wLOlrr71W9R2e5eOVVz/7ls/O5rrme/mMYupMRKDTxnkzlq1OrVUjz+fnPhUgjuuMg2347KoHmgOAuXPnxjTP+HmwqO9+97sxb8cdd4xpVyS0qy1SpM5s5CArrJ5xeFUo1UaxL3B7xbzwwgsAsn7BM+0ehKWdzhCuRcoWPLvdE1u4D7Et/HxPIOsXvoLh/QIA/O53v4vpp59+uge/oj04+OCDY9rbq2HDhr3nd7iN4gB0rhxwFQ4AHH744THNig63V+rMdaBzVaPsbRTbJ6V+8hVwINvfeJ//mc98JuZxIKjzzjsvpr2/4OBGp556aky7UqSstvDfzUHnVndMy+nRo0dX3feuu+6quj8gv6iXLVgVyNfi9sjtwu0Z903f/OY3M2UqA6kxLddfPftuDnr64IMPAsi+n4wbNy6mixzTamVbCCGEEEIIIYSoM3rZFkIIIYQQQggh6kzDZOQub+LAJhw0yyUfvLn9/PPPj2k/yzMlRwCAe+65B0BW3vfUU0/FdCoY2rx585JlcckBSw/aiZR0mX+/S2U4gIMHTgE6ZeJ5tnApGtuSpWYclMhhGQ7f1wOy+D3LANvC/YVlNn4uLZNnC6/r1DmGQFaK45x99tnJz7o/pLYUtCspv2BbuAQWyAauS+FtUJ7fsRTKP8MSWt4e4P5yyy23dONXtAcpW/CWipQt8uRhLjXLs0XKL9iHODiO+0Ut+7cT3Ab4mddXXXVVzGOJoH82r41yv+Bz0vfee++Y5m0v++67L4CsLdgfvb8oaxvlQUn5+eQ2ZOrUqQCy9cdjMrfFkiVLYh5L/tmGl156KYBOqWbX63qg2XYMLptHqu/Oa6Nq1UuqjXL7db2XB3/k+uctRn7OsWzRaQve5thdW3D98piV+4vhw4cDyPbnHJhQtnhvv6jVj9bqu3lrmduApeUsSfdxRBF9t1a2hRBCCCGEEEKIOqOXbSGEEEIIIYQQos40TEbuUcZZOnDUUUdVfY5lSizT8LMkFy9eHPNYKuYydJe0AdnopxwpNXXuI0uX+bzVdiQlI+fIf/53lpfxZ7fbbjsA6ajYQKfclSM3ckRTlvSvXLmy6vss31++fPl7/ZSWJ2ULjnbtsCSJJTPTpk0D0HkGdldSUR45yjz7WEpKU3a/SNkizy/8rHmWKTEuceUoqf369Yvp008/Pab9bFu2NUfInjlzJoD2lS6nbMHS5FptVHdtkYrEDNT2C5cCAu1/akXKFhy11+siFf0X6LQFb4Ng/LlmW/K5quwjqW0BHFU41Z+0E7X84oc//GHmc0A2Mvlxxx0HINsf8/dvv/12AMDNN98c8ziqMssxH3/88arvX3/99THtUvZ2lcvWsoWT10Z96lOfApDvF6n+4rnnnotpP+ccSPf/7Bdl7Ltr2YL7VrcF9xd8LbcFj2l5Ox+f9nLBBRdU3Ze3QZbRFo2AWV2HAAANcUlEQVQYR+X13UOHDo1pl/3z9ZtlTKuVbSGEEEIIIYQQos40bGV7woQJALJn+PKsgq9I550r6+cF86oor2L7yizPgvPMB69yOxxEpN1XUBmf8eTZIJ6x87rMmyX0ICz8d06PHDkSQDZQAac5cJrD9c+B7dodVwfwzB/P+PkKkis7AOCEE06IaQ+AdsUVV8Q8fu533313AMDs2bNjns8Mdk3799gv/Bz0MuAqgDxbnHLKKQCyK50emAZItzH8ffchvj6vPrBSxGdkp0+fHvM4yEq7rmg7bgtud7gufWUuFQwQAPr37191Tf6sB7Th/iTPLxz2C+/PykDKFjvssENMP/PMMwCAww47LOYNHDgwpn0FKK+/cH/gIEO8cs59h1PWNspXQV0pCNT2i/nz58e0t2Gpdh8AJk2aBCBra74Xt1c+TpgyZUrMW7RoUY9+TyvjtuB2p5Yt+FlO9ReM1/tNN90U81LqHqasY9re+EVvbDFixIiY98ILLyTv5T7yxBNPxDwOZNquSg+n1jjK2/a+6LvdR7hdYrsUiVa2hRBCCCGEEEKIOqOXbSGEEEIIIYQQos40TEa+/fbbA8g/F9bl3ywD5wAGLknIOzfVv8fSAg6gxkFsPFgBX79MuMzi1VdfjXmc9nobMGBAzOOgAmyjFC5lYumIB/ICsoE9/Fp5dm133BYsC+a0S4u5zlm2d8wxx2Q+1xWXe/LfOYgQ+0UtH2t3PNATB+jjtJ89ywFrOPCJB0nJO//RA3dwAA/eUjF+/PiYdolbSipYBtwWtfyCt8JwMECXW7ItuC5dVsjSZbYl+4Xboux+wfXv7Q7QWZf8/PI5zb5dLK+/9a0Y3N98+9vfjmm2RbsHQKuF9xcsleS+OdVfcNvvMvG8Z9kl51znp556akyzdNb7i7K2USlbsES4li1SQaMY/96CBQti3gMPPBDTc+bMiWkfv5W1jVpdv+iuLfhs7h//+McxzdJnl0TzltgyUWsc5YHP8vru7r7r8ZYB7rv5fPmlS5cCyPbzzUI53z6FEEIIIYQQQogGopdtIYQQQgghhBCizhSme3DJB0s/egPLRDiiKUs/u95TZHG5YG8jHrvEkM8e5MjxLFl/rzzRKb9hGQ5zzjnndOs6Dz30UEzzGavsFy4H5IizopMZM2bU5TrcLrEtUvWeOidd1PaL7pJnC94C48gWaVw6mzpftidwf8F2lS26z+qOozzCOG914WulbKFxVJp6jWlvu+22mOZTY1K2UN+dpl624C2QvGWA2yOXkc+bNy/52bLTiL6b3/u22Wabqs824/YjrWwLIYQQQgghhBB1puV39PPMVd4sVlkDevQ1ecGNGNmib8gLVsHIFn1Dd2zhyCaNRbZoHtRfNA/dGUeJvuHGG29M5qfaK/lHY2FfmDVrVvIzvuIqWzSW7vTdbi8OnN0saGVbCCGEEEIIIYSoM3rZFkIIIYQQQggh6kzLy8i7w+puzBf1g8/HE8XC59KLYpEtmgfZonmQLZoHjaOaB9mieZAtmgc+C73Z0Mq2EEIIIYQQQghRZ/SyLYQQQgghhBBC1Bm9bAshhBBCCCGEEHVGL9tCCCGEEEIIIUSd0cu2EEIIIYQQQghRZ/SyLYQQQgghhBBC1Bm9bAshhBBCCCGEEHXGQgjd/7DZKwCebVxx2oZNQgj9G3kD2aJHNNQeskWPkC2aB9mieZAtmgfZonmQLZoH2aJ5kC2ah27Zokcv20IIIYQQQgghhKiNZORCCCGEEEIIIUSd0cu2EEIIIYQQQghRZ1ryZdvMNjazn5rZKjP7k5nNNbMhRZerjJjZRmZ2lZn9j5n91cyCmX2i6HKVDTMbY2a3mdmzZvY3M3vazCab2UeLLlvZMLN9zOxeM3vJzN42sxVm9mMzG1502QRgZv/V0U5dWHRZyoaZ7dFR913/vVl02cqKmY0ys/vN7C8d46nfm9leRZerTJjZfTl+Eczsv4ouX9kws8+a2Z1mtrLDJx4xs6OLLlcZMbM9zeyBjnHt62Z2k5kNLLpcPeUDRRegp5jZWgDuBfA2gP8DIAC4EMB/m9nWIYS3iixfCRkG4EAADwP4vwBGFluc0nImgOUAvglgBYDtAHwbwJ5m9pkQwr8KLFvZ+Bgq/nANgFcADAFwFoAFZjYihKDAIwVhZocA2KbocgiMB/A7+v93iipImTGz4wFM7/g3EZUFmG0BrFVkuUrIiQDW6ZK3C4CpAOb1fXHKi5ltDeBuAAsA/AeAvwIYA2C2ma0ZQri2yPKVCTPbDcCdAH4N4AAA66PyvnePmW0fQni7yPL1hJYLkGZmp6DSAG0eQljckfdJAH8E8I0QwtQiy1c2zOx9/iJnZscCuB7AJ0MIywotWMkws/4hhFe65B0B4HsAPh9CuLeYkgkAMLPNATwF4MwQwneLLk8ZMbN1UbHBaQBuAXBRCOHcYktVLsxsDwD/DeALIYS7Cy5OqelQoC0CcHYI4YpiSyO6YmazARwGYFAI4fWiy1MWzGwSKosXHwsh/IXyFwAIIYRdCitcyTCzuwF8AsC/hRDe6cj7NICHAIwLIVxTYPF6RCvKyL8MYIG/aANACGEpgN8AGF1YqUqKVkybg64v2h34ytGGfVkWkeS1jv/+s9BSlJspAJ4IIdxadEGEaAKOBvAvADOKLojIYmYfBvA1AP+pF+0+Zw1U+um/dcl/E635ztTK7AzgLn/RBoAQwu9QGU/9e2Gl6gWt+OBsCeDxRP4TALQnUohOdu/476JCS1FSzOz9ZraGmX0KwEwALwH4YcHFKiVmtiuAI1CRa4riudnM3jWz18zsFsVcKYRdUVF6HGxmS8zsHTNbbGbjii6YwFcBfBQVZZroW+Z0/PdKMxtsZuua2X8A+DyAy4srVil5F8A/EvlvA9iqj8uyWrTcnm1U9kO+kch/HcB6fVwWIZoSM9sQwAUA7g4h/L7o8pSUBwFs35FeDGCvEMLKAstTSszsg6hMdlwWQni66PKUnFUAvgtgPoA/oRJb4psA/sfMtpN/9CmDO/5diooNlqCymjrdzD4QQphWZOFKzhEAVgK4o+iClI0QwuMd211uR+fk7D8BjA0haLK8b3kaldXtiJltAmAQWkwl2Iov20AlKFpXrM9LIUQTYmZrA/g5KkGHjiq4OGXmcFSC3gxFZQ/YXWa2q+IZ9DkTAHwYwEVFF6TshBD+F8D/UtZ8M7sflT144wFoD33f8T5UVk+PDCHM7ci7t2Mv99lmdmVotaA+bYCZDQawN4BpLJ8VfUOHEu02VNSyY1GRk48GMMPM/h5CuLnI8pWMaQB+0HFyyJWoLLZeh8r2l5bawtqKMvI3UKnwrqyH9Iq3EKXBzD6ESvTSoQD2CSGsKLhIpSWEsCiE8GDHHuHPA1gblajkoo/okCefA+A8AGt2SALX7fiz///7iyuhCCE8AuAPAD5ddFlKhseRuKtL/p0ABqKyeiT6nsNQGZtLQl4Mk1BZNf1iCOEXIYR7QgjjAfwYwDQza8X3ppakY2LjQgBnAHgZwJMAngfwKwAvFli0HtOKD80TqOzb7spwVAwhRCnpkMveBmBHAKNCCI8VXCTRQQjhTVSk5MOKLkvJGArgQwB+gMpkrP8DKmqDNwCMKKZogjCkFWuicTyRk+8qwZZaOWojjgCwMISwsOiClJQRqNR/V5nyQ6gcPTWg74tUXkII5wH4OICtUYnMfwiATwF4oNCC9ZBWfNmeB2BnMxvqGR2yp89C5xGKktIx23ozKiuoo0MICwoukiDMbCCAf0NlX6ToO/4fgD0T/4DKC/ieqEyCiIIwsx0AbIZKjAPRd9ze8d99uuTvA2BFCOGlPi5P6enwhS2hVe0ieQnAtma2Rpf8nQD8HZX4UKIPCSG8FUJ4LITwspnti8pYqqVOUWjFPdvXAzgJwM/N7FxUZsMnAngOlSA4oo8xszEdSQ8GtZ+ZvQLglRDC/IKKVTauRiW4zUUA3jIzDiqxQnLyvsPMbgfwCIBHUQkCtRkqZzu/g0pwKNFHdCgK7uuab2YA8GwIoepvonGY2c0AlqLiH2+iEiDtbFSkgVcVWLQy8itUzjyfaWYfB/AMgDEARkKxPoriCFT6iVuKLkiJmQ7gJwD+08yuQWXP9pcBHALg8hBCKjq2aABmth2A/VDpL4DKCQpfBzAlhPDbwgrWC6wV41907MO7HMAXUJE83QPgVAUeKgYzy3uI5ocQ9ujLspQVM1sGYJOcP38nhPDtvitNuTGzCQAOBLApKmd2PofKC99ktVHNQUebdVEIQQG5+hAzOxuVQesmANZCZRXpDgDfCiG01B68dsDM1gEwGZWX7PVQOQrs4hCCXvb6mI5tYC8AWBBC+FLR5SkzZrYfKoE1t0RlG9ISVAJzzQwhvFtk2cqEmW2JyiLqVgDWROUY26tCCDcWWrBe0JIv20IIIYQQQgghRDPTinu2hRBCCCGEEEKIpkYv20IIIYQQQgghRJ3Ry7YQQgghhBBCCFFn9LIthBBCCCGEEELUGb1sCyGEEEIIIYQQdUYv20IIIYQQQgghRJ3Ry7YQQgghhBBCCFFn9LIthBBCCCGEEELUGb1sCyGEEEIIIYQQdeb/AwSpwd3JDSDVAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# standardize the data\n",
"mean_vals = np.mean(my_digits_X, axis=0)\n",
"std_val = np.std(my_digits_X)\n",
"my_digits_X_standardized = (my_digits_X - mean_vals)/std_val\n",
"\n",
"# obtain predictions for test set\n",
"my_digits_y_preds = cnn.predict(my_digits_X_standardized)\n",
"\n",
"# print model's accuracy on the input data\n",
"print('Test Accuracy: %.2f%%' % \\\n",
" (100*np.sum(my_digits_y == my_digits_y_preds)/len(my_digits_y)))\n",
"\n",
"# plot the results\n",
"print('\\nClassification Results: (predicted class is shown below image)')\n",
"fix, ax = plt.subplots(nrows=1, ncols=10, \n",
" sharex=True, sharey=True, \n",
" figsize=(14, 8))\n",
"ax = ax.flatten()\n",
"for i in range(10):\n",
" img = my_digits_X_standardized[:10][i].reshape(28, 28)\n",
" ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
" ax[i].set_xlabel(my_digits_y_preds[:10][i])\n",
"ax[0].set_xticks([])\n",
"ax[0].set_yticks([])\n",
"plt.tight_layout()\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this case our CNN yielded 100% accuracy! \n",
"\n",
"In general there several techniques for assessing how well a model generalizes to unseen data, such as through learning curves."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***\n",
" \n",
"## Closing Remarks\n",
"\n",
"In this tutorial, we saw how to use TensorFlow's low-level API to define a convolutional neural network for handwritten digit recognition. By now you should have a basic understanding of how TensorFlow works and how to use it for training your own models. \n",
"\n",
"More recently, the use of placeholders and feed dictionaries has been superceded by the newer `Dataset` API, which allows us to utilize the same input pipeline across all of our models. Complementary to this are the introduction of new methods for defining feature columns and custom networks (estimators). You can read more about these new paradigms in the following blog posts by the Google team:\n",
"1. https://developers.googleblog.com/2017/09/introducing-tensorflow-datasets.html\n",
"2. https://developers.googleblog.com/2017/11/introducing-tensorflow-feature-columns.html\n",
"3. https://developers.googleblog.com/2017/12/creating-custom-estimators-in-tensorflow.html\n",
"\n",
"Personally, I've found it faster to use **Keras** for building my models. This cuts down on a lot of cumbersome \"boilerplate\" code and makes easier to quickly go from idea to experiment. Knowing the lower-level TensorFlow API is nonetheless useful for debugging and can be used to implement more advanced or customized features that exist in TensorFlow but are not part of the Keras frontend. \n",
"\n",
"For a primer on how to use Keras for rapidly prototyping more complex networks, see my tutorial here . "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}