{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "YMiI5CDX5JZ0" }, "source": [ "# 2. kNN for (Q)SPR modeling ⚛️\n", "\n", "\"Open" ] }, { "cell_type": "markdown", "metadata": { "id": "dlaMhdbLXofj" }, "source": [ "## Goals of this exercise 🌟" ] }, { "cell_type": "markdown", "metadata": { "id": "YCWB0IvbXp4O" }, "source": [ "* We will learn how to construct a simple SPR model using kNN\n", "* We will learn the importance of data normalization (scaling)\n", "* We will review the concepts of training and test split and cross-validation\n", "* We will review some of the performance metrics for assesing classification models" ] }, { "cell_type": "markdown", "metadata": { "id": "XI_fHNSEYck6" }, "source": [ "## A quick reminder ✅" ] }, { "cell_type": "markdown", "metadata": { "id": "8wst0cbF39zU" }, "source": [ "Probably the simplest data-driven model that you can think of is k-nearest neighbours (kNN). It simply predicts future data as the average (or mode) of the \"k\" nearest neighbours of the queried point.\n", "\n", "As simple as this idea might be, it works relatively good in various applications. One of them is the generation of (quantitative) structure-property relationships ((Q)SPR) models {cite}`yuan2019developing, shen2003development`. Whether the word \"Quantitative\" is sometimes included or not depends on whether the model in question is a regression model or a classification model. Do you remember the difference?\n", "\n", "The key question in kNN is what do we consider a neighbour and what not? This indicates us that we need to define a sort of similarity or distance metric that allows us to distinguish neighbouring points from points that are far away. \n", "\n", "Common distance metrics use the different [mathematical norms](https://en.wikipedia.org/wiki/Norm_(mathematics)). For example, the Euclidean distance:\n", "\n", "$$\n", " d(\\textbf{x}, \\textbf{x'}) = \\sqrt{\\sum_i^D (x_i - x'_i)^2}\n", "$$\n", "\n", "```{figure} media/02_kNN/kNN.png\n", ":alt: kNN\n", ":width: 75%\n", ":align: center\n", "\n", "Among the k-nearest neighbours (k=5), the majority of points are red 1s. Therefore, the queried point (green x) will be labeled as \"red 1\". Image taken from {cite}`murphy2022probabilistic`. \n", "```\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "1sWnwvt_FjQa" }, "source": [ "Let's exemplify the use of kNN by constructing a SPR model that predicts the whether a molecule is mutagenic or not. \n", "\n", "Mutagenicity is the property of substances to induce genetic mutation. It is one of the most important environmental, health and safety (EHS) properties to check when dealing with novel chemicals (e.g., drugs or solvents). In this case, we are going to use the data of mutagenicity on Salmonella typhimurium (Ames test). This dataset was collected by the [Istituto di Ricerche Farmacologiche Mario Negri](https://www.marionegri.it/), merging experimental data from a benchmark dataset\n", "compiled by {cite}`hansen2009benchmark` from a collection of data made available\n", "by the [Japan Health Ministry](https://www.nihs.go.jp/dgm/amesqsar.html) within their Ames (Q)SAR project." ] }, { "cell_type": "markdown", "metadata": { "id": "R9-iQHe-XZm6" }, "source": [ "Let's fist import some libraries" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "id": "V3C3ZNc8XjsA" }, "outputs": [], "source": [ "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import seaborn as sn\n", "import numpy as np\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": { "id": "oWmicwjH5Ksm" }, "source": [ "## Get data 📚\n", "\n", "We have previously computed some molecular descriptors that will serve as input to our model. However, this is also an important step to consider when facing a problem like this: what are the important inputs to model mutagenicity? how do we know if these pre-computed features are enough for modeling mutagenicity? can we generate relevant molecular features automatically? 🤔\n", "\n", "Ok, let's use [pandas](https://pandas.pydata.org/) to import the data as a DataFrame..." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 679 }, "id": "vn-yAIhU5ILt", "outputId": "a468e405-9061-4763-8ece-0b8223d03486" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0IdCASSMILESStatusExperimental valuePredicted valueNumValenceElectronsqedTPSAMolMRBalabanJBertzCTMolWtMolLogP
001100-00-5O=[N+]([O-])c1ccc(cc1)ClTraining11520.46360243.1438.10643.003401244.429658157.5562.24820
112100-01-6O=[N+]([O-])c1ccc(N)cc1Training11520.35954469.1637.50883.003401242.429658138.1261.17700
223100-02-7O=[N+]([O-])c1ccc(O)cc1Training01520.47072863.3734.76123.003401241.674771139.1101.30040
334100-11-8O=[N+]([O-])c1ccc(cc1)CBrTraining10580.43258643.1445.72742.913802257.648013216.0342.48970
445100-12-9O=[N+]([O-])c1ccc(cc1)CCTraining00580.47978543.1442.47442.913802253.299498151.1652.15720
................................................
57595759576720395-16-8O=C1N(C(=O)N(C(=O)N1CC=C)CC2OC2)CC=CTraining101020.48509078.5369.35602.668492627.435628265.269-1.05750
57605760576834718-47-3O=C(C(Br)(Br)Br)ClTraining11420.49598717.0740.37203.79111885.425922315.1862.59030
57615761576943204-63-3N(CCBr)CCBrTraining11440.56855612.0340.45772.44747328.870765230.9311.36580
57625762577052583-35-4N#Cc2cc(cc(c2(N=Nc1ccc(cc1(NC(=O)C))N(CCOC)CCO...Training111840.264581185.59125.35252.5529771169.342047485.4573.84768
576357635771188021-38-7O(c1c(cc(cc1Br)C)C(C)(C)C)CC=CTraining10880.7353929.2373.27103.263016388.457194283.2094.61982
\n", "

5764 rows × 15 columns

\n", "
" ], "text/plain": [ " Unnamed: 0 Id CAS \\\n", "0 0 1 100-00-5 \n", "1 1 2 100-01-6 \n", "2 2 3 100-02-7 \n", "3 3 4 100-11-8 \n", "4 4 5 100-12-9 \n", "... ... ... ... \n", "5759 5759 5767 20395-16-8 \n", "5760 5760 5768 34718-47-3 \n", "5761 5761 5769 43204-63-3 \n", "5762 5762 5770 52583-35-4 \n", "5763 5763 5771 188021-38-7 \n", "\n", " SMILES Status \\\n", "0 O=[N+]([O-])c1ccc(cc1)Cl Training \n", "1 O=[N+]([O-])c1ccc(N)cc1 Training \n", "2 O=[N+]([O-])c1ccc(O)cc1 Training \n", "3 O=[N+]([O-])c1ccc(cc1)CBr Training \n", "4 O=[N+]([O-])c1ccc(cc1)CC Training \n", "... ... ... \n", "5759 O=C1N(C(=O)N(C(=O)N1CC=C)CC2OC2)CC=C Training \n", "5760 O=C(C(Br)(Br)Br)Cl Training \n", "5761 N(CCBr)CCBr Training \n", "5762 N#Cc2cc(cc(c2(N=Nc1ccc(cc1(NC(=O)C))N(CCOC)CCO... Training \n", "5763 O(c1c(cc(cc1Br)C)C(C)(C)C)CC=C Training \n", "\n", " Experimental value Predicted value NumValenceElectrons qed \\\n", "0 1 1 52 0.463602 \n", "1 1 1 52 0.359544 \n", "2 0 1 52 0.470728 \n", "3 1 0 58 0.432586 \n", "4 0 0 58 0.479785 \n", "... ... ... ... ... \n", "5759 1 0 102 0.485090 \n", "5760 1 1 42 0.495987 \n", "5761 1 1 44 0.568556 \n", "5762 1 1 184 0.264581 \n", "5763 1 0 88 0.735392 \n", "\n", " TPSA MolMR BalabanJ BertzCT MolWt MolLogP \n", "0 43.14 38.1064 3.003401 244.429658 157.556 2.24820 \n", "1 69.16 37.5088 3.003401 242.429658 138.126 1.17700 \n", "2 63.37 34.7612 3.003401 241.674771 139.110 1.30040 \n", "3 43.14 45.7274 2.913802 257.648013 216.034 2.48970 \n", "4 43.14 42.4744 2.913802 253.299498 151.165 2.15720 \n", "... ... ... ... ... ... ... \n", "5759 78.53 69.3560 2.668492 627.435628 265.269 -1.05750 \n", "5760 17.07 40.3720 3.791118 85.425922 315.186 2.59030 \n", "5761 12.03 40.4577 2.447473 28.870765 230.931 1.36580 \n", "5762 185.59 125.3525 2.552977 1169.342047 485.457 3.84768 \n", "5763 9.23 73.2710 3.263016 388.457194 283.209 4.61982 \n", "\n", "[5764 rows x 15 columns]" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "if 'google.colab' in str(get_ipython()):\n", " df = pd.read_csv(\"https://raw.githubusercontent.com/edgarsmdn/MLCE_book/main/references/mutagenicity_kNN.csv\")\n", "else:\n", " df = pd.read_csv(\"references/mutagenicity_kNN.csv\")\n", "\n", "df" ] }, { "cell_type": "markdown", "metadata": { "id": "ttVnS2-dbxcy" }, "source": [ "The library pandas has many useful functions for data analytics. For example, we can print the type of the data we have..." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "IyH-W0ty8-Oi", "outputId": "7706cfa4-ad96-4a52-e546-7cc83c005161" }, "outputs": [ { "data": { "text/plain": [ "Unnamed: 0 int64\n", "Id int64\n", "CAS object\n", "SMILES object\n", "Status object\n", "Experimental value int64\n", "Predicted value object\n", "NumValenceElectrons int64\n", "qed float64\n", "TPSA float64\n", "MolMR float64\n", "BalabanJ float64\n", "BertzCT float64\n", "MolWt float64\n", "MolLogP float64\n", "dtype: object" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.dtypes " ] }, { "cell_type": "markdown", "metadata": { "id": "v7PaIMXTcKxk" }, "source": [ "And have a look at the first rows of our data to see how it looks like" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 206 }, "id": "GBvQkpwi6vCV", "outputId": "bd1aca5e-5c9d-45fc-a399-bc7c54f482d9" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0IdCASSMILESStatusExperimental valuePredicted valueNumValenceElectronsqedTPSAMolMRBalabanJBertzCTMolWtMolLogP
001100-00-5O=[N+]([O-])c1ccc(cc1)ClTraining11520.46360243.1438.10643.003401244.429658157.5562.2482
112100-01-6O=[N+]([O-])c1ccc(N)cc1Training11520.35954469.1637.50883.003401242.429658138.1261.1770
223100-02-7O=[N+]([O-])c1ccc(O)cc1Training01520.47072863.3734.76123.003401241.674771139.1101.3004
334100-11-8O=[N+]([O-])c1ccc(cc1)CBrTraining10580.43258643.1445.72742.913802257.648013216.0342.4897
445100-12-9O=[N+]([O-])c1ccc(cc1)CCTraining00580.47978543.1442.47442.913802253.299498151.1652.1572
\n", "
" ], "text/plain": [ " Unnamed: 0 Id CAS SMILES Status \\\n", "0 0 1 100-00-5 O=[N+]([O-])c1ccc(cc1)Cl Training \n", "1 1 2 100-01-6 O=[N+]([O-])c1ccc(N)cc1 Training \n", "2 2 3 100-02-7 O=[N+]([O-])c1ccc(O)cc1 Training \n", "3 3 4 100-11-8 O=[N+]([O-])c1ccc(cc1)CBr Training \n", "4 4 5 100-12-9 O=[N+]([O-])c1ccc(cc1)CC Training \n", "\n", " Experimental value Predicted value NumValenceElectrons qed TPSA \\\n", "0 1 1 52 0.463602 43.14 \n", "1 1 1 52 0.359544 69.16 \n", "2 0 1 52 0.470728 63.37 \n", "3 1 0 58 0.432586 43.14 \n", "4 0 0 58 0.479785 43.14 \n", "\n", " MolMR BalabanJ BertzCT MolWt MolLogP \n", "0 38.1064 3.003401 244.429658 157.556 2.2482 \n", "1 37.5088 3.003401 242.429658 138.126 1.1770 \n", "2 34.7612 3.003401 241.674771 139.110 1.3004 \n", "3 45.7274 2.913802 257.648013 216.034 2.4897 \n", "4 42.4744 2.913802 253.299498 151.165 2.1572 " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head()" ] }, { "cell_type": "markdown", "metadata": { "id": "Eb8STE1Gc1qN" }, "source": [ "and access rows by index" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 175 }, "id": "F-vLO_KHeUEy", "outputId": "8d1c9199-93b7-4f76-a0e0-fb035b7f9c56", "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0IdCASSMILESStatusExperimental valuePredicted valueNumValenceElectronsqedTPSAMolMRBalabanJBertzCTMolWtMolLogP
001100-00-5O=[N+]([O-])c1ccc(cc1)ClTraining11520.46360243.1438.10643.003401244.429658157.5562.2482
112100-01-6O=[N+]([O-])c1ccc(N)cc1Training11520.35954469.1637.50883.003401242.429658138.1261.1770
223100-02-7O=[N+]([O-])c1ccc(O)cc1Training01520.47072863.3734.76123.003401241.674771139.1101.3004
334100-11-8O=[N+]([O-])c1ccc(cc1)CBrTraining10580.43258643.1445.72742.913802257.648013216.0342.4897
\n", "
" ], "text/plain": [ " Unnamed: 0 Id CAS SMILES Status \\\n", "0 0 1 100-00-5 O=[N+]([O-])c1ccc(cc1)Cl Training \n", "1 1 2 100-01-6 O=[N+]([O-])c1ccc(N)cc1 Training \n", "2 2 3 100-02-7 O=[N+]([O-])c1ccc(O)cc1 Training \n", "3 3 4 100-11-8 O=[N+]([O-])c1ccc(cc1)CBr Training \n", "\n", " Experimental value Predicted value NumValenceElectrons qed TPSA \\\n", "0 1 1 52 0.463602 43.14 \n", "1 1 1 52 0.359544 69.16 \n", "2 0 1 52 0.470728 63.37 \n", "3 1 0 58 0.432586 43.14 \n", "\n", " MolMR BalabanJ BertzCT MolWt MolLogP \n", "0 38.1064 3.003401 244.429658 157.556 2.2482 \n", "1 37.5088 3.003401 242.429658 138.126 1.1770 \n", "2 34.7612 3.003401 241.674771 139.110 1.3004 \n", "3 45.7274 2.913802 257.648013 216.034 2.4897 " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "first_rows = df.iloc[:4]\n", "first_rows" ] }, { "cell_type": "markdown", "metadata": { "id": "ejwwO7ETcTPw" }, "source": [ "We can access columns in the DataFrame by the column's name" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "4domchYBdi-f", "outputId": "967c3ff1-4905-442a-be5e-2a84bb89c037" }, "outputs": [ { "data": { "text/plain": [ "0 1\n", "1 1\n", "2 0\n", "3 1\n", "4 0\n", " ..\n", "5759 1\n", "5760 1\n", "5761 1\n", "5762 1\n", "5763 1\n", "Name: Experimental value, Length: 5764, dtype: int64" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y_experimental = df['Experimental value']\n", "y_experimental" ] }, { "cell_type": "markdown", "metadata": { "id": "Ro4VEmtzei8w" }, "source": [ "If we would like to get the subset of data that is labeled as mutagenic (i.e., 'Experimental value' equal to 1), we could do it like this" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 679 }, "id": "p_1NdUU2gVhn", "outputId": "10082daf-42b4-4fa9-b260-3e2292777f0f" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0IdCASSMILESStatusExperimental valuePredicted valueNumValenceElectronsqedTPSAMolMRBalabanJBertzCTMolWtMolLogP
001100-00-5O=[N+]([O-])c1ccc(cc1)ClTraining11520.46360243.1438.10643.003401244.429658157.5562.24820
112100-01-6O=[N+]([O-])c1ccc(N)cc1Training11520.35954469.1637.50883.003401242.429658138.1261.17700
334100-11-8O=[N+]([O-])c1ccc(cc1)CBrTraining10580.43258643.1445.72742.913802257.648013216.0342.48970
667100-13-0O=[N+]([O-])c1ccc(C=C)cc1Training10560.47766043.1443.18743.000887276.648462149.1492.23780
778100-14-1O=[N+]([O-])c1ccc(cc1)CClTraining11580.38948243.1442.65342.913802257.648013171.5832.33360
................................................
57595759576720395-16-8O=C1N(C(=O)N(C(=O)N1CC=C)CC2OC2)CC=CTraining101020.48509078.5369.35602.668492627.435628265.269-1.05750
57605760576834718-47-3O=C(C(Br)(Br)Br)ClTraining11420.49598717.0740.37203.79111885.425922315.1862.59030
57615761576943204-63-3N(CCBr)CCBrTraining11440.56855612.0340.45772.44747328.870765230.9311.36580
57625762577052583-35-4N#Cc2cc(cc(c2(N=Nc1ccc(cc1(NC(=O)C))N(CCOC)CCO...Training111840.264581185.59125.35252.5529771169.342047485.4573.84768
576357635771188021-38-7O(c1c(cc(cc1Br)C)C(C)(C)C)CC=CTraining10880.7353929.2373.27103.263016388.457194283.2094.61982
\n", "

3251 rows × 15 columns

\n", "
" ], "text/plain": [ " Unnamed: 0 Id CAS \\\n", "0 0 1 100-00-5 \n", "1 1 2 100-01-6 \n", "3 3 4 100-11-8 \n", "6 6 7 100-13-0 \n", "7 7 8 100-14-1 \n", "... ... ... ... \n", "5759 5759 5767 20395-16-8 \n", "5760 5760 5768 34718-47-3 \n", "5761 5761 5769 43204-63-3 \n", "5762 5762 5770 52583-35-4 \n", "5763 5763 5771 188021-38-7 \n", "\n", " SMILES Status \\\n", "0 O=[N+]([O-])c1ccc(cc1)Cl Training \n", "1 O=[N+]([O-])c1ccc(N)cc1 Training \n", "3 O=[N+]([O-])c1ccc(cc1)CBr Training \n", "6 O=[N+]([O-])c1ccc(C=C)cc1 Training \n", "7 O=[N+]([O-])c1ccc(cc1)CCl Training \n", "... ... ... \n", "5759 O=C1N(C(=O)N(C(=O)N1CC=C)CC2OC2)CC=C Training \n", "5760 O=C(C(Br)(Br)Br)Cl Training \n", "5761 N(CCBr)CCBr Training \n", "5762 N#Cc2cc(cc(c2(N=Nc1ccc(cc1(NC(=O)C))N(CCOC)CCO... Training \n", "5763 O(c1c(cc(cc1Br)C)C(C)(C)C)CC=C Training \n", "\n", " Experimental value Predicted value NumValenceElectrons qed \\\n", "0 1 1 52 0.463602 \n", "1 1 1 52 0.359544 \n", "3 1 0 58 0.432586 \n", "6 1 0 56 0.477660 \n", "7 1 1 58 0.389482 \n", "... ... ... ... ... \n", "5759 1 0 102 0.485090 \n", "5760 1 1 42 0.495987 \n", "5761 1 1 44 0.568556 \n", "5762 1 1 184 0.264581 \n", "5763 1 0 88 0.735392 \n", "\n", " TPSA MolMR BalabanJ BertzCT MolWt MolLogP \n", "0 43.14 38.1064 3.003401 244.429658 157.556 2.24820 \n", "1 69.16 37.5088 3.003401 242.429658 138.126 1.17700 \n", "3 43.14 45.7274 2.913802 257.648013 216.034 2.48970 \n", "6 43.14 43.1874 3.000887 276.648462 149.149 2.23780 \n", "7 43.14 42.6534 2.913802 257.648013 171.583 2.33360 \n", "... ... ... ... ... ... ... \n", "5759 78.53 69.3560 2.668492 627.435628 265.269 -1.05750 \n", "5760 17.07 40.3720 3.791118 85.425922 315.186 2.59030 \n", "5761 12.03 40.4577 2.447473 28.870765 230.931 1.36580 \n", "5762 185.59 125.3525 2.552977 1169.342047 485.457 3.84768 \n", "5763 9.23 73.2710 3.263016 388.457194 283.209 4.61982 \n", "\n", "[3251 rows x 15 columns]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mutagenic_data = df[df['Experimental value']==1]\n", "mutagenic_data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's have a look at the predictions by VEGA which we'll use to compare our results with:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 1\n", "1 1\n", "2 1\n", "3 0\n", "4 0\n", " ..\n", "5759 0\n", "5760 1\n", "5761 1\n", "5762 1\n", "5763 0\n", "Name: Predicted value, Length: 5764, dtype: object" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['Predicted value']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We notice the data type says `object` even though it should be all `0` and `1`. Let's try to find out why:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 1\n", "3 0\n", "1844 Non Predicted\n", "Name: Predicted value, dtype: object" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['Predicted value'].drop_duplicates()" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0IdCASSMILESStatusExperimental valuePredicted valueNumValenceElectronsqedTPSAMolMRBalabanJBertzCTMolWtMolLogP
18441844184616709-86-7C=C[Si](C)(C)CClTraining0Non Predicted420.4014390.0038.3993.57547168.480406134.6822.19800
2194219421972179-59-1C=CCSSCCCTraining0Non Predicted480.3338390.0045.4042.61629352.490225148.2962.96380
411841184125624-92-0CSSCTraining0Non Predicted260.4528400.0027.0301.9747456.00000094.2041.62740
4180418041876317-18-6N#CSCSC#NTraining0Non Predicted360.32159547.5831.2752.76838695.083765130.1971.37246
5633563356417783-54-2FN(F)FTraining1Non Predicted260.3839803.245.1632.3237908.00000071.0010.94190
565556555663676-83-5CP(Cl)ClTraining1Non Predicted260.4269900.0024.5502.32379010.754888116.9152.40570
\n", "
" ], "text/plain": [ " Unnamed: 0 Id CAS SMILES Status \\\n", "1844 1844 1846 16709-86-7 C=C[Si](C)(C)CCl Training \n", "2194 2194 2197 2179-59-1 C=CCSSCCC Training \n", "4118 4118 4125 624-92-0 CSSC Training \n", "4180 4180 4187 6317-18-6 N#CSCSC#N Training \n", "5633 5633 5641 7783-54-2 FN(F)F Training \n", "5655 5655 5663 676-83-5 CP(Cl)Cl Training \n", "\n", " Experimental value Predicted value NumValenceElectrons qed \\\n", "1844 0 Non Predicted 42 0.401439 \n", "2194 0 Non Predicted 48 0.333839 \n", "4118 0 Non Predicted 26 0.452840 \n", "4180 0 Non Predicted 36 0.321595 \n", "5633 1 Non Predicted 26 0.383980 \n", "5655 1 Non Predicted 26 0.426990 \n", "\n", " TPSA MolMR BalabanJ BertzCT MolWt MolLogP \n", "1844 0.00 38.399 3.575471 68.480406 134.682 2.19800 \n", "2194 0.00 45.404 2.616293 52.490225 148.296 2.96380 \n", "4118 0.00 27.030 1.974745 6.000000 94.204 1.62740 \n", "4180 47.58 31.275 2.768386 95.083765 130.197 1.37246 \n", "5633 3.24 5.163 2.323790 8.000000 71.001 0.94190 \n", "5655 0.00 24.550 2.323790 10.754888 116.915 2.40570 " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[ (df['Predicted value'] != \"0\") & (df['Predicted value'] != \"1\") ]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For some inputs the VEGA model decides to output `non predicted`. As detailed in the documentation, it does this for particularly uncertain predictions. Let's remove these from the data set:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "non_predicted = df[ (df['Predicted value'] != \"0\") & (df['Predicted value'] != \"1\") ].index\n", "df_clean = df.drop(non_predicted)" ] }, { "cell_type": "markdown", "metadata": { "id": "Si1CwO1Ol9-r" }, "source": [ "Let's now collect all the input features of our dataset and remove the unnecessary ones:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 424 }, "id": "IahR4BJimCJ2", "outputId": "d2e62d3c-e82e-4944-c496-8370a07c5a76" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NumValenceElectronsqedTPSAMolMRBalabanJBertzCTMolWtMolLogP
0520.46360243.1438.10643.003401244.429658157.5562.24820
1520.35954469.1637.50883.003401242.429658138.1261.17700
2520.47072863.3734.76123.003401241.674771139.1101.30040
3580.43258643.1445.72742.913802257.648013216.0342.48970
4580.47978543.1442.47442.913802253.299498151.1652.15720
...........................
57591020.48509078.5369.35602.668492627.435628265.269-1.05750
5760420.49598717.0740.37203.79111885.425922315.1862.59030
5761440.56855612.0340.45772.44747328.870765230.9311.36580
57621840.264581185.59125.35252.5529771169.342047485.4573.84768
5763880.7353929.2373.27103.263016388.457194283.2094.61982
\n", "

5758 rows × 8 columns

\n", "
" ], "text/plain": [ " NumValenceElectrons qed TPSA MolMR BalabanJ BertzCT \\\n", "0 52 0.463602 43.14 38.1064 3.003401 244.429658 \n", "1 52 0.359544 69.16 37.5088 3.003401 242.429658 \n", "2 52 0.470728 63.37 34.7612 3.003401 241.674771 \n", "3 58 0.432586 43.14 45.7274 2.913802 257.648013 \n", "4 58 0.479785 43.14 42.4744 2.913802 253.299498 \n", "... ... ... ... ... ... ... \n", "5759 102 0.485090 78.53 69.3560 2.668492 627.435628 \n", "5760 42 0.495987 17.07 40.3720 3.791118 85.425922 \n", "5761 44 0.568556 12.03 40.4577 2.447473 28.870765 \n", "5762 184 0.264581 185.59 125.3525 2.552977 1169.342047 \n", "5763 88 0.735392 9.23 73.2710 3.263016 388.457194 \n", "\n", " MolWt MolLogP \n", "0 157.556 2.24820 \n", "1 138.126 1.17700 \n", "2 139.110 1.30040 \n", "3 216.034 2.48970 \n", "4 151.165 2.15720 \n", "... ... ... \n", "5759 265.269 -1.05750 \n", "5760 315.186 2.59030 \n", "5761 230.931 1.36580 \n", "5762 485.457 3.84768 \n", "5763 283.209 4.61982 \n", "\n", "[5758 rows x 8 columns]" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X = df_clean.drop(['Unnamed: 0', 'Id','CAS','SMILES','Status','Experimental value','Predicted value'],axis=1)\n", "X" ] }, { "cell_type": "markdown", "metadata": { "id": "ei-EOGnle97C" }, "source": [ "### Exercise - manipulate a DataFrame ❗❗\n", "\n", "* How many molecules in our dataset have a `qed` less than 0.5?\n", "* What is the molecule with the largest molecular weight `MolWt`?\n", "* What is the average number of valance electrons `NumValenceElectrons` of the molecules in our dataset? " ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "id": "DwAoulsggXUJ" }, "outputs": [], "source": [ "# Your code here" ] }, { "cell_type": "markdown", "metadata": { "id": "h2uOyvKYB_QE" }, "source": [ "## Training and test split ✂️\n", "\n", "Remember, our goal is to create a model that **generalize well** to unseen data and not simply fit some seen data perfectly. To achieve this, our goal while splitting the data should be to ensure that the distribution of the test set is as close as possible to the expected distribution of the future data. Hence, often what we want is to make the training and test datasets to have a similar distribution.\n", "\n", "For example, let's see if how many molecules in our dataset are mutagenic vs. non-mutagenic." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "QUCpGIgcB0y6", "outputId": "82ed55b9-820f-4539-9cb4-8a827906003e" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Percentage of mutagenic molecules : 56.42584230635638\n", "Percentage of non-mutagenic molecules: 43.57415769364362\n" ] } ], "source": [ "y = df_clean['Experimental value'].to_numpy()\n", "perc_mutagenic = y.sum()/len(y)*100\n", "print('Percentage of mutagenic molecules :', perc_mutagenic)\n", "print('Percentage of non-mutagenic molecules:', 100-perc_mutagenic)" ] }, { "cell_type": "markdown", "metadata": { "id": "YFlMSb6TCZ7S" }, "source": [ "In this case, the proportion of mutagenic and non-mutagenic data is very similar. Then, we will go ahead and split the data randomly. However, when you have a much more imbalanced dataset, how would you split the data better? You can look for instance at what [stratified splitting](https://scikit-learn.org/stable/modules/cross_validation.html#stratification) is and why is it important.\n", "\n", "In summary, when splitting your data, always think about the distribution of your splits!" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "id": "DQR8ATRjCPCq" }, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "id": "I09X-mSqCRtN" }, "outputs": [], "source": [ "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "yeIZ2XI5jwoD", "outputId": "32dc2318-453f-4baa-9b3b-285cac037c8b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training points: 4606\n", "Training points: 1152\n" ] } ], "source": [ "print('Training points: ', len(y_train))\n", "print('Training points: ', len(y_test))" ] }, { "cell_type": "markdown", "metadata": { "id": "fnRg4cMqDZia" }, "source": [ "### Exercise - splits distribution ❗❗\n", "\n", "* Check what is the proportion of mutagenic molecules in your train and test set? Was the random splitting good?" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "id": "outc17O_DVKZ" }, "outputs": [], "source": [ "# Your code here" ] }, { "cell_type": "markdown", "metadata": { "id": "KGvYC2i5hW_A" }, "source": [ "## Feature scaling 📏" ] }, { "cell_type": "markdown", "metadata": { "id": "iTfo1fwH7qMv" }, "source": [ "It is always a good practice to scale your data before starting modeling. This helps the training process of many machine learning models. This is specially true for kNN which works on distances! The distance between two points is naturally afected by the dimensions of the input space. Look for example at the Euclidean distance, if one dimension ranges from 0 to 10,000 and another ranges from 0 to 1, the former one will potentially impact the distance value much more!\n", "\n", "We do not want this to happen. Therefore, we need to scale all input features in order to give the same importance to all dimensions regardless of their original scale.\n", "\n", "Here, we will use the method known as [standardization](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler). Here, we move the distribution of the data to have unit variance and a mean equal to zero.\n", "\n", "$$\n", "\\hat{\\textbf{x}} = \\frac{\\textbf{x}-\\mu_x}{\\sigma_x}\n", "$$\n", "\n", "Of course there are other [scaling methods](https://medium.com/greyatom/why-how-and-when-to-scale-your-features-4b30ab09db5e) are available and you might want to review them and check which one is better than the other in which conditions." ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "id": "KlCTYKzV7Lja" }, "outputs": [], "source": [ "from sklearn.preprocessing import StandardScaler" ] }, { "cell_type": "markdown", "metadata": { "id": "3nd3K52ulXb1" }, "source": [ "We initialize our scaler function" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "id": "JDrRFz3o7ORI" }, "outputs": [], "source": [ "scaler = StandardScaler()" ] }, { "cell_type": "markdown", "metadata": { "id": "sPkVGBadlbgO" }, "source": [ "and fit it to our data (i.e., get the mean vector $\\mu_x$ and the standard deviation vector $\\sigma_x$." ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "j6tTHY4Al5U-", "outputId": "8e2cff7e-8754-4ecd-c82f-0b3a55e99138" }, "outputs": [ { "data": { "text/plain": [ "StandardScaler()" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "scaler.fit(X_train)" ] }, { "cell_type": "markdown", "metadata": { "id": "hAMOgj-JprOW" }, "source": [ "Now, let's scale our data" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "m4TAblST7lIL", "outputId": "ca653b6e-0605-47f1-ec43-d752cf4ebe13" }, "outputs": [ { "data": { "text/plain": [ "array([[ 6.91725031, -2.50154035, 6.01891181, ..., 3.43592554,\n", " 6.12598659, -1.84190261],\n", " [-0.39909237, -1.36770143, 0.74694415, ..., -0.14444052,\n", " -0.41184123, -0.84860786],\n", " [ 0.34652217, 0.01697949, -0.31275131, ..., 0.18125274,\n", " 0.22156128, 0.73178752],\n", " ...,\n", " [ 0.29992126, 2.01678401, 0.34399709, ..., 0.69463898,\n", " 0.66401444, -0.23869147],\n", " [ 0.53292581, -0.04711992, -0.74958466, ..., 0.09406034,\n", " 0.34165461, 1.34128367],\n", " [ 1.83775125, -2.00364986, 0.47608443, ..., 1.57463088,\n", " 6.0606014 , 2.62264517]])" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_train = scaler.transform(X_train)\n", "X_test = scaler.transform(X_test)\n", "X_train" ] }, { "cell_type": "markdown", "metadata": { "id": "g0zIoG5op95Z" }, "source": [ "### Exercise - read documentation ❗❗\n", "\n", "* What are exactly the mean an standard deviation vectors that we used to scaled the data? Go to the [sklearn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) and learn how you can access these. Reading the documentation of a library is a super important skill to learn! " ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "id": "8WDOvs_WqVFj" }, "outputs": [], "source": [ "# Your code here" ] }, { "cell_type": "markdown", "metadata": { "id": "Ti0h11a7CjPs" }, "source": [ "## kNN model 🏘️" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "id": "fXHhnxgICgJD" }, "outputs": [], "source": [ "from sklearn.neighbors import KNeighborsClassifier" ] }, { "cell_type": "markdown", "metadata": { "id": "axfX0cUMEWJN" }, "source": [ "We initialize the kNN model by specifying the parameter \"k\". Later, we will review some ways that help us in determining this parameter better. For now, let's set it to 3." ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "id": "0C18Ne_tCxdn" }, "outputs": [], "source": [ "knn = KNeighborsClassifier(n_neighbors=3)" ] }, { "cell_type": "markdown", "metadata": { "id": "Ny_zGEKvEwf5" }, "source": [ "we train it" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "WYtljFtOCzB7", "outputId": "6d8e7c6c-4233-4f71-8752-1e7c4f0372f7" }, "outputs": [ { "data": { "text/plain": [ "KNeighborsClassifier(n_neighbors=3)" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "knn.fit(X_train, y_train)" ] }, { "cell_type": "markdown", "metadata": { "id": "J1UYc51DEz3w" }, "source": [ "we predict the test set" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "id": "8op0R9jbC4R0" }, "outputs": [], "source": [ "y_pred = knn.predict(X_test)" ] }, { "cell_type": "markdown", "metadata": { "id": "M2EDTPLAHWit" }, "source": [ "Let's now evaluate our kNN model!\n", "\n", "We will use several metrics for classification, for a quick reminder on them check the [documentation](https://scikit-learn.org/stable/modules/classes.html#classification-metrics)." ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "id": "1iStpf04HV_1" }, "outputs": [], "source": [ "from sklearn.metrics import (confusion_matrix, accuracy_score, precision_score, \n", " recall_score, f1_score, roc_auc_score, ConfusionMatrixDisplay) " ] }, { "cell_type": "markdown", "metadata": { "id": "T6ppJg7yGM2R" }, "source": [ "Let first look at the confusion matrix" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "47uV_7PAg808", "outputId": "2b304c3a-788c-4d04-8e05-87a58e77337d" }, "outputs": [ { "data": { "text/plain": [ "array([[327, 176],\n", " [139, 510]], dtype=int64)" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cm = confusion_matrix(y_test, y_pred)\n", "cm" ] }, { "cell_type": "markdown", "metadata": { "id": "CD9QYwXEGQAl" }, "source": [ "to see a prettier confusion matrix" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 296 }, "id": "zxE7tmAQGaxn", "outputId": "d77df237-8655-48d4-9c0d-d4906b1cf61c" }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfsAAAGwCAYAAACuFMx9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA+NklEQVR4nO3deXxU1f3/8fdkX2cgQDJEQhSRJRJAUWFaRSxIgFSx4K9VEaJFrBioglCkorIo8YutCzaCdUMrFFdoQVwiShAJVsEIZUkNoomGSVBKQoLZZu7vD5rREZAMk4XMfT19nMcjc++5937GRx588jnn3HsthmEYAgAAASuotQMAAADNi2QPAECAI9kDABDgSPYAAAQ4kj0AAAGOZA8AQIAj2QMAEOBCWjsAf7jdbpWUlCg2NlYWi6W1wwEA+MgwDB0+fFiJiYkKCmq++rO6ulq1tbV+nycsLEwRERFNEFHLatPJvqSkRElJSa0dBgDAT8XFxerSpUuznLu6ulpnJcfIWeby+1x2u1379u1rcwm/TSf72NhYSVKXeXMU1Mb+xwON1WNpaWuHADSbenetNnz5hOff8+ZQW1srZ5lLX249U9bYUx89qDjsVvKAL1RbW0uyb0kNQ/dBEREkewSskKDw1g4BaHYtMRUbE2tRTOypX8ettjtd3KaTPQAAjeUy3HL58TYYl+FuumBaGMkeAGAKbhly69SzvT/HtjZuvQMAIMCR7AEApuBugv98MXfuXFksFq/Wq1cvz/7q6mplZmaqQ4cOiomJ0dixY1Va6r0gt6ioSOnp6YqKilJ8fLxmzpyp+vp6n787w/gAAFNwGYZcxqkPxZ/Kseeee67eeecdz+eQkO/T7rRp0/T666/r5Zdfls1m05QpUzRmzBh98MEHR6/ncik9PV12u12bN2/W/v37NWHCBIWGhmrhwoU+xUGyBwCgmYSEhMhutx+zvby8XE8//bRWrFihX/ziF5KkZ599Vr1799aWLVs0aNAgvf3229q1a5feeecdJSQkqH///lqwYIFmzZqluXPnKiwsrNFxMIwPADCFhgV6/jRJqqio8Go1NTUnvOZnn32mxMREdevWTePGjVNRUZEkaevWraqrq9OwYcM8fXv16qWuXbsqLy9PkpSXl6fU1FQlJCR4+qSlpamiokI7d+706buT7AEApuCWIZcfrSHZJyUlyWazeVpWVtZxrzdw4EAtW7ZMb775ppYsWaJ9+/bpkksu0eHDh+V0OhUWFqZ27dp5HZOQkCCn0ylJcjqdXom+YX/DPl8wjA8AgA+Ki4tltVo9n8PDj//gq5EjR3p+7tu3rwYOHKjk5GS99NJLioyMbPY4f4jKHgBgCk01jG+1Wr3aiZL9j7Vr1049evRQYWGh7Ha7amtrdejQIa8+paWlnjl+u91+zOr8hs/HWwfwU0j2AABTaFiN70/zR2Vlpfbu3avOnTtrwIABCg0N1fr16z37CwoKVFRUJIfDIUlyOBzasWOHysrKPH1ycnJktVqVkpLi07UZxgcAoBnMmDFDV1xxhZKTk1VSUqJ7771XwcHBuvbaa2Wz2TRx4kRNnz5dcXFxslqtmjp1qhwOhwYNGiRJGj58uFJSUjR+/HgtWrRITqdTc+bMUWZmZqNHExqQ7AEApuD+X/PneF989dVXuvbaa/Xtt9+qU6dOuvjii7VlyxZ16tRJkvTwww8rKChIY8eOVU1NjdLS0vT44497jg8ODtbatWs1efJkORwORUdHKyMjQ/Pnz/c5dpI9AMAUGlbV+3O8L1auXPmT+yMiIpSdna3s7OwT9klOTta6det8uu7xkOwBAKbgMuTnW++aLpaWxgI9AAACHJU9AMAUWnrO/nRCsgcAmIJbFrlk8ev4tophfAAAAhyVPQDAFNzG0ebP8W0VyR4AYAouP4fx/Tm2tTGMDwBAgKOyBwCYgpkre5I9AMAU3IZFbsOP1fh+HNvaGMYHACDAUdkDAEyBYXwAAAKcS0Fy+TGg7WrCWFoayR4AYAqGn3P2BnP2AADgdEVlDwAwBebsAQAIcC4jSC7Djzn7Nvy4XIbxAQAIcFT2AABTcMsitx81rlttt7Qn2QMATMHMc/YM4wMAEOCo7AEApuD/Aj2G8QEAOK0dnbP340U4DOMDAIDTFZU9AMAU3H4+G5/V+AAAnOaYswcAIMC5FWTa++yZswcAIMBR2QMATMFlWOTy4zW1/hzb2kj2AABTcPm5QM/FMD4AADhdUdkDAEzBbQTJ7cdqfDer8QEAOL0xjA8AAAIWlT0AwBTc8m9FvbvpQmlxJHsAgCn4/1CdtjsY3nYjBwAAjUJlDwAwBf+fjd9262OSPQDAFHifPQAAAa6hsvennaoHHnhAFotFt99+u2fbkCFDZLFYvNott9zidVxRUZHS09MVFRWl+Ph4zZw5U/X19T5fn8oeAIBm9NFHH+mJJ55Q3759j9k3adIkzZ8/3/M5KirK87PL5VJ6errsdrs2b96s/fv3a8KECQoNDdXChQt9ioHKHgBgCg0P1fGn+aqyslLjxo3Tk08+qfbt2x+zPyoqSna73dOsVqtn39tvv61du3bphRdeUP/+/TVy5EgtWLBA2dnZqq2t9SkOkj0AwBTchsXvJkkVFRVeraam5oTXzMzMVHp6uoYNG3bc/cuXL1fHjh3Vp08fzZ49W0eOHPHsy8vLU2pqqhISEjzb0tLSVFFRoZ07d/r03RnGBwDAB0lJSV6f7733Xs2dO/eYfitXrtS2bdv00UcfHfc81113nZKTk5WYmKjt27dr1qxZKigo0GuvvSZJcjqdXolekuez0+n0KWaSPQDAFNx+Phu/4aE6xcXFXsPt4eHhx/QtLi7WbbfdppycHEVERBz3fDfffLPn59TUVHXu3FlDhw7V3r17dfbZZ59ynMfDMD4AwBQa3nrnT5Mkq9Xq1Y6X7Ldu3aqysjKdf/75CgkJUUhIiHJzc7V48WKFhITI5XIdc8zAgQMlSYWFhZIku92u0tJSrz4Nn+12u0/fnWQPAEATGzp0qHbs2KH8/HxPu+CCCzRu3Djl5+crODj4mGPy8/MlSZ07d5YkORwO7dixQ2VlZZ4+OTk5slqtSklJ8SkehvEBAKbgkkUuPx6M48uxsbGx6tOnj9e26OhodejQQX369NHevXu1YsUKjRo1Sh06dND27ds1bdo0DR482HOL3vDhw5WSkqLx48dr0aJFcjqdmjNnjjIzM487mvBTSPYAAFP44VD8qR7fVMLCwvTOO+/okUceUVVVlZKSkjR27FjNmTPH0yc4OFhr167V5MmT5XA4FB0drYyMDK/78huLZA8AQAvYsGGD5+ekpCTl5uae9Jjk5GStW7fO72uT7AEApuCSb0Pxxzu+rSLZAwBM4XQaxm9pJHsAgCmY+RW3bTdyAADQKFT2AABTMPx8n73Rht9nT7IHAJgCw/gAACBgUdkDAEzhh6+pPdXj2yqSPQDAFFx+vvXOn2NbW9uNHAAANAqVPQDAFBjGBwAgwLkVJLcfA9r+HNva2m7kAACgUajsAQCm4DIscvkxFO/Psa2NZA8AMAXm7AEACHCGn2+9M3iCHgAAOF1R2QMATMEli1x+vMzGn2NbG8keAGAKbsO/eXe30YTBtDCG8QEACHBU9iZn3eSU7YMyhR6skSTV2iN1MO0MHUlpr6CqesW9WayoPeUKOVQjV3SoqlLjdHBUF7kjj/7qxH5YpoS/f37cc+9bMECu2NAW+y7AiZzb7xuNva5Q3XseUoeONVow+yJteb+zZ//rm/5x3OOezk7Ra38/x/P5QodT195YoDPPrlBdbbB2fNJB9/1xYLPHj6bh9nOBnj/HtjaSvcnVtwvXt1ckqa5ThGRIsR8dUOen/6PiGamSpJDyOn0zOlm19kiFHqxRp5f3KaSiVs4be0iSKs/rqCO923mdM37FXgXVuUn0OG1ERLq0r9CmnNe7as7Cj47Zf/2VaV6fBwwq1W135mtzbqJn288uLdHvZ+XruSd669NtnRQc7FZyt8PNHjuajlsWuf2Yd/fn2NZ2WiT77OxsPfjgg3I6nerXr58ee+wxXXTRRa0dlikc6dPe6/PB9K6yfVCq8C8rdXhQvJy/7eHZV98xQt+mJ8n+t0LJZUjBFhlhQXKFhXn6BFXWKeqzCpVd063FvgNwMlu3JGjrloQT7v/vwQivz4Mudmr7to5ylkRLkoKC3frdbTv0TPa5evv1ZE+/4i+szRMw0MRafUzixRdf1PTp03Xvvfdq27Zt6tevn9LS0lRWVtbaoZmP21DMtm8UVONW9Zkxx+0S/J1L7ohgKfj4f+FaPzogd2iQKvt1aM5IgWbTrn21LvxZqVdS796jXB3jq+U2pMXPbNDfVr+peX/KU/JZFa0YKXzV8AQ9f1pb1erJ/qGHHtKkSZN04403KiUlRUuXLlVUVJSeeeaZ1g7NNMJKjqjbH/6ls2d8qE4v7dP+iT1UZ486pl9QZZ3av/2Vyn8Wf8JzWbccUOWAjjLCWv1XCzglQ0cW67sjIdqc+/2cvj2xSpI07rcFWvlcD82bNUiVh0OV9dgHiomtba1Q4aOGOXt/WlvVqpHX1tZq69atGjZsmGdbUFCQhg0bpry8vGP619TUqKKiwqvBf7XxESqe2VdfTeujip8nKGH5XoU6j3j1sVTXK/Gve1SbEKmDI7oc9zwR+w4rrPQ7VQzq1BJhA83i8vQibXi7i+pqgz3bLP/7l/LF53toc26iCgva6eGF50mGdPEvSlopUqDxWjXZf/PNN3K5XEpI8J5LS0hIkNPpPKZ/VlaWbDabpyUlJbVUqIEtJEh1nSJUkxSjb6/oqpozotQu9/v//5ZqlxKX7pE7IljOiT2l4OP/2li3lKnmjCjVJB1/CgA43Z3b91slJVfqrbXJXtv/+024JKnoi1jPtvq6YDn3Ryk+wfsPY5y+3LJ4no9/Sq0NL9BrU2MSs2fPVnl5uacVFxe3dkiByZAs9W5JRyv6M5bsloIt2n9TTxmhx/+VsdS4FJP/rSoGnXiIHzjdDf/ll/psj037Cm1e2z8raKfamiB1Sar0bAsOdive/p3KnMdOeeH0ZPxvNf6pNqMNJ/tWXY3fsWNHBQcHq7S01Gt7aWmp7Hb7Mf3Dw8MVHh7eUuGZQoc1RapKaaf6dmEKqnErdus3iiysUMktvf6X6PfIUuuWc3wPBVW7pGqXJMkVEyoFff+LH/PJt5Lb0OEBHVvrqwAnFBFZr8Qzqjyf7Z2PqFv3ch0+HKoDpUeTdWRUnS6+rERP/eXcY47/7kio1v3jTI2buEcHyiJV5ozU2OsKJUmb3ks8pj9OT7z1rpWEhYVpwIABWr9+va666ipJktvt1vr16zVlypTWDM00givrlPBCoUIq6uSKDFZtYpRKbuml73q2U+Rn5Yr48mglc+Z9+V7HfXF3f9V3+P52JeuWMlX1jZM76rS4mxPwck6vQ3rgsQ88nyf9/t+SpHfWJenhhedLki4d9rVkkXLfOf6alGeyz5XbZdEdd29TeLhLBbva64+3/UyVh8OO2x84nVgMw2jVp/2++OKLysjI0BNPPKGLLrpIjzzyiF566SXt2bPnmLn8H6uoqJDNZlPX/7tPQRERP9kXaKt6LT52/QoQKOrdNXpn32MqLy+X1do8zy1oyBW/yrlRodGn/sdZXVWtVl3+bLPG2lxavQz7zW9+owMHDuiee+6R0+lU//799eabb5400QMA4AuG8VvZlClTGLYHAKCZnBbJHgCA5saz8QEACHBmHsZvU/fZAwAA31HZAwBMwcyVPckeAGAKZk72DOMDABDgSPYAAFPw6yU4fo4KPPDAA7JYLLr99ts926qrq5WZmakOHTooJiZGY8eOPebx8UVFRUpPT1dUVJTi4+M1c+ZM1dfX+3x9kj0AwBQMyc8X4Zyajz76SE888YT69u3rtX3atGlas2aNXn75ZeXm5qqkpERjxozx7He5XEpPT1dtba02b96s5557TsuWLdM999zjcwwkewCAKbRGZV9ZWalx48bpySefVPv27T3by8vL9fTTT+uhhx7SL37xCw0YMEDPPvusNm/erC1btkiS3n77be3atUsvvPCC+vfvr5EjR2rBggXKzs5WbW2tT3GQ7AEA8EFFRYVXq6mpOWHfzMxMpaena9iwYV7bt27dqrq6Oq/tvXr1UteuXZWXlydJysvLU2pqqtfj49PS0lRRUaGdO3f6FDPJHgBgCk1V2SclJclms3laVlbWca+3cuVKbdu27bj7nU6nwsLC1K5dO6/tCQkJcjqdnj4/fk9Mw+eGPo3FrXcAAFNoqlvviouLvd56Fx4efkzf4uJi3XbbbcrJyVHEafBWVip7AAB8YLVavdrxkv3WrVtVVlam888/XyEhIQoJCVFubq4WL16skJAQJSQkqLa2VocOHfI6rrS0VHa7XZJkt9uPWZ3f8LmhT2OR7AEAptCSC/SGDh2qHTt2KD8/39MuuOACjRs3zvNzaGio1q9f7zmmoKBARUVFcjgckiSHw6EdO3aorKzM0ycnJ0dWq1UpKSk+fXeG8QEApmAYFhl+DOP7cmxsbKz69OnjtS06OlodOnTwbJ84caKmT5+uuLg4Wa1WTZ06VQ6HQ4MGDZIkDR8+XCkpKRo/frwWLVokp9OpOXPmKDMz87ijCT+FZA8AQCt4+OGHFRQUpLFjx6qmpkZpaWl6/PHHPfuDg4O1du1aTZ48WQ6HQ9HR0crIyND8+fN9vhbJHgBgCq39PvsNGzZ4fY6IiFB2drays7NPeExycrLWrVvn13Ulkj0AwCR4EQ4AAAhYVPYAAFNoyQV6pxuSPQDAFMw8jE+yBwCYgpkre+bsAQAIcFT2AABTMPwcxm/LlT3JHgBgCoYkw/Dv+LaKYXwAAAIclT0AwBTcssjSik/Qa00kewCAKbAaHwAABCwqewCAKbgNiyw8VAcAgMBlGH6uxm/Dy/EZxgcAIMBR2QMATMHMC/RI9gAAUyDZAwAQ4My8QI85ewAAAhyVPQDAFMy8Gp9kDwAwhaPJ3p85+yYMpoUxjA8AQICjsgcAmAKr8QEACHCG/HsnfRsexWcYHwCAQEdlDwAwBYbxAQAIdCYexyfZAwDMwc/KXm24smfOHgCAAEdlDwAwBZ6gBwBAgDPzAj2G8QEACHBU9gAAczAs/i2ya8OVPckeAGAKZp6zZxgfAIAAR2UPADAHHqoDAEBgM/Nq/EYl+3/+85+NPuGVV155ysEAAICm16hkf9VVVzXqZBaLRS6Xy594AABoPm14KN4fjVqg53a7G9VI9ACA01XDML4/zRdLlixR3759ZbVaZbVa5XA49MYbb3j2DxkyRBaLxavdcsstXucoKipSenq6oqKiFB8fr5kzZ6q+vt7n7+7XnH11dbUiIiL8OQUAAC2jhRfodenSRQ888IDOOeccGYah5557TqNHj9Ynn3yic889V5I0adIkzZ8/33NMVFSU52eXy6X09HTZ7XZt3rxZ+/fv14QJExQaGqqFCxf6FIvPt965XC4tWLBAZ5xxhmJiYvT5559Lku6++249/fTTvp4OAICAdMUVV2jUqFE655xz1KNHD91///2KiYnRli1bPH2ioqJkt9s9zWq1eva9/fbb2rVrl1544QX1799fI0eO1IIFC5Sdna3a2lqfYvE52d9///1atmyZFi1apLCwMM/2Pn366KmnnvL1dAAAtBBLEzSpoqLCq9XU1Jz0yi6XSytXrlRVVZUcDodn+/Lly9WxY0f16dNHs2fP1pEjRzz78vLylJqaqoSEBM+2tLQ0VVRUaOfOnT59c5+H8Z9//nn99a9/1dChQ73mFvr166c9e/b4ejoAAFpGEw3jJyUleW2+9957NXfu3OMesmPHDjkcDlVXVysmJkarVq1SSkqKJOm6665TcnKyEhMTtX37ds2aNUsFBQV67bXXJElOp9Mr0UvyfHY6nT6F7nOy//rrr9W9e/djtrvdbtXV1fl6OgAA2pTi4mKv4fbw8PAT9u3Zs6fy8/NVXl6uV155RRkZGcrNzVVKSopuvvlmT7/U1FR17txZQ4cO1d69e3X22Wc3acw+D+OnpKTo/fffP2b7K6+8ovPOO69JggIAoMkZTdAkz+r6hvZTyT4sLEzdu3fXgAEDlJWVpX79+unRRx89bt+BAwdKkgoLCyVJdrtdpaWlXn0aPtvtdp++us+V/T333KOMjAx9/fXXcrvdeu2111RQUKDnn39ea9eu9fV0AAC0jNPgrXdut/uEc/z5+fmSpM6dO0uSHA6H7r//fpWVlSk+Pl6SlJOTI6vV6pkKaCyfk/3o0aO1Zs0azZ8/X9HR0brnnnt0/vnna82aNbr88st9PR0AAAFp9uzZGjlypLp27arDhw9rxYoV2rBhg9566y3t3btXK1as0KhRo9ShQwdt375d06ZN0+DBg9W3b19J0vDhw5WSkqLx48dr0aJFcjqdmjNnjjIzM39yNOF4Tuk++0suuUQ5OTmncigAAK2ipV9xW1ZWpgkTJmj//v2y2Wzq27ev3nrrLV1++eUqLi7WO++8o0ceeURVVVVKSkrS2LFjNWfOHM/xwcHBWrt2rSZPniyHw6Ho6GhlZGR43ZffWKf8UJ2PP/5Yu3fvlnR0Hn/AgAGneioAAJpfCz9U56eePZOUlKTc3NyTniM5OVnr1q3z7cLH4XOy/+qrr3Tttdfqgw8+ULt27SRJhw4d0s9+9jOtXLlSXbp08TsoAADQdHxejX/TTTeprq5Ou3fv1sGDB3Xw4EHt3r1bbrdbN910U3PECACA/xoW6PnT2iifK/vc3Fxt3rxZPXv29Gzr2bOnHnvsMV1yySVNGhwAAE3FYhxt/hzfVvmc7JOSko778ByXy6XExMQmCQoAgCbXwnP2pxOfh/EffPBBTZ06VR9//LFn28cff6zbbrtNf/rTn5o0OAAA4L9GVfbt27eXxfL9XEVVVZUGDhyokJCjh9fX1yskJES//e1vddVVVzVLoAAA+OU0eKhOa2lUsn/kkUeaOQwAAJqZiYfxG5XsMzIymjsOAADQTE75oTqSVF1drdraWq9tP3wTEAAApw0TV/Y+L9CrqqrSlClTFB8fr+joaLVv396rAQBwWmqit961RT4n+z/84Q969913tWTJEoWHh+upp57SvHnzlJiYqOeff745YgQAAH7weRh/zZo1ev755zVkyBDdeOONuuSSS9S9e3clJydr+fLlGjduXHPECQCAf0y8Gt/nyv7gwYPq1q2bpKPz8wcPHpQkXXzxxdq4cWPTRgcAQBNpeIKeP62t8jnZd+vWTfv27ZMk9erVSy+99JKkoxV/w4txAADA6cPnZH/jjTfq008/lSTdeeedys7OVkREhKZNm6aZM2c2eYAAADQJEy/Q83nOftq0aZ6fhw0bpj179mjr1q3q3r27+vbt26TBAQAA//l1n70kJScnKzk5uSliAQCg2Vjk51vvmiySlteoZL948eJGn/D3v//9KQcDAACaXqOS/cMPP9yok1ksllZJ9t1mfaQQS2iLXxdoCa+X5Ld2CECzqTjsVvseLXQxE99616hk37D6HgCANovH5QIAgEDl9wI9AADaBBNX9iR7AIAp+PsUPFM9QQ8AALQtVPYAAHMw8TD+KVX277//vq6//no5HA59/fXXkqS//e1v2rRpU5MGBwBAkzHx43J9Tvavvvqq0tLSFBkZqU8++UQ1NTWSpPLyci1cuLDJAwQAAP7xOdnfd999Wrp0qZ588kmFhn7/IJuf//zn2rZtW5MGBwBAUzHzK259nrMvKCjQ4MGDj9lus9l06NChpogJAICmZ+In6Plc2dvtdhUWFh6zfdOmTerWrVuTBAUAQJNjzr7xJk2apNtuu00ffvihLBaLSkpKtHz5cs2YMUOTJ09ujhgBAIAffB7Gv/POO+V2uzV06FAdOXJEgwcPVnh4uGbMmKGpU6c2R4wAAPjNzA/V8TnZWywW3XXXXZo5c6YKCwtVWVmplJQUxcTENEd8AAA0DRPfZ3/KD9UJCwtTSkpKU8YCAACagc/J/rLLLpPFcuIVie+++65fAQEA0Cz8vX3OTJV9//79vT7X1dUpPz9f//73v5WRkdFUcQEA0LQYxm+8hx9++Ljb586dq8rKSr8DAgAATavJ3np3/fXX65lnnmmq0wEA0LRMfJ99k731Li8vTxEREU11OgAAmpSZb73zubIfM2aMV/vVr36lQYMG6cYbb9Tvfve75ogRAIA2Z8mSJerbt6+sVqusVqscDofeeOMNz/7q6mplZmaqQ4cOiomJ0dixY1VaWup1jqKiIqWnpysqKkrx8fGaOXOm6uvrfY7F58reZrN5fQ4KClLPnj01f/58DR8+3OcAAAAIRF26dNEDDzygc845R4Zh6LnnntPo0aP1ySef6Nxzz9W0adP0+uuv6+WXX5bNZtOUKVM0ZswYffDBB5Ikl8ul9PR02e12bd68Wfv379eECRMUGhrq81tmLYZhNHpgwuVy6YMPPlBqaqrat2/v27duBhUVFbLZbBqi0QqxhJ78AKANeqskv7VDAJpNxWG32vf4XOXl5bJarc1zjf/lirNnL1SwH9PNrupq7c36o4qLi71iDQ8PV3h4eKPOERcXpwcffFBXX321OnXqpBUrVujqq6+WJO3Zs0e9e/dWXl6eBg0apDfeeEO//OUvVVJSooSEBEnS0qVLNWvWLB04cEBhYWGNjt2nYfzg4GANHz6ct9sBANqcpnrFbVJSkmw2m6dlZWWd9Noul0srV65UVVWVHA6Htm7dqrq6Og0bNszTp1evXuratavy8vIkHV0Ll5qa6kn0kpSWlqaKigrt3LnTp+/u8zB+nz599Pnnn+uss87y9VAAANq841X2J7Jjxw45HA5VV1crJiZGq1atUkpKivLz8xUWFqZ27dp59U9ISJDT6ZQkOZ1Or0TfsL9hny98Tvb33XefZsyYoQULFmjAgAGKjo722t9cwzAAAPitCVbUNyy4a4yePXsqPz9f5eXleuWVV5SRkaHc3Fz/g/BRo5P9/Pnzdccdd2jUqFGSpCuvvNLrsbmGYchiscjlcjV9lAAA+KsVnqAXFham7t27S5IGDBigjz76SI8++qh+85vfqLa2VocOHfKq7ktLS2W32yVJdrtd//rXv7zO17Bav6FPYzU62c+bN0+33HKL3nvvPZ8uAAAAjnK73aqpqdGAAQMUGhqq9evXa+zYsZKkgoICFRUVyeFwSJIcDofuv/9+lZWVKT4+XpKUk5Mjq9Xq84voGp3sGxbtX3rppT5dAACA00FLP1Rn9uzZGjlypLp27arDhw9rxYoV2rBhg9566y3ZbDZNnDhR06dPV1xcnKxWq6ZOnSqHw6FBgwZJkoYPH66UlBSNHz9eixYtktPp1Jw5c5SZmdno1f8NfJqz/6m33QEAcFpr4WH8srIyTZgwQfv375fNZlPfvn311ltv6fLLL5d09F0zQUFBGjt2rGpqapSWlqbHH3/cc3xwcLDWrl2ryZMny+FwKDo6WhkZGZo/f77PofuU7Hv06HHShH/w4EGfgwAAINA8/fTTP7k/IiJC2dnZys7OPmGf5ORkrVu3zu9YfEr28+bNO+YJegAAtAVmfja+T8n+mmuu8SwSAACgTTHx++wb/QQ95usBAGibfF6NDwBAm2Tiyr7Ryd7tdjdnHAAANCvm7AEACHQmrux9eusdAABoe6jsAQDmYOLKnmQPADAFM8/ZM4wPAECAo7IHAJgDw/gAAAQ2hvEBAEDAorIHAJgDw/gAAAQ4Eyd7hvEBAAhwVPYAAFOw/K/5c3xbRbIHAJiDiYfxSfYAAFPg1jsAABCwqOwBAObAMD4AACbQhhO2PxjGBwAgwFHZAwBMwcwL9Ej2AABzMPGcPcP4AAAEOCp7AIApMIwPAECgYxgfAAAEKip7AIApMIwPAECgM/EwPskeAGAOJk72zNkDABDgqOwBAKbAnD0AAIGOYXwAABCoqOwBAKZgMQxZjFMvz/05trWR7AEA5sAwPgAACFQkewCAKTSsxven+SIrK0sXXnihYmNjFR8fr6uuukoFBQVefYYMGSKLxeLVbrnlFq8+RUVFSk9PV1RUlOLj4zVz5kzV19f7FAvD+AAAc2jhYfzc3FxlZmbqwgsvVH19vf74xz9q+PDh2rVrl6Kjoz39Jk2apPnz53s+R0VFeX52uVxKT0+X3W7X5s2btX//fk2YMEGhoaFauHBho2Mh2QMA4IOKigqvz+Hh4QoPDz+m35tvvun1edmyZYqPj9fWrVs1ePBgz/aoqCjZ7fbjXuvtt9/Wrl279M477yghIUH9+/fXggULNGvWLM2dO1dhYWGNiplhfACAKTTVMH5SUpJsNpunZWVlNer65eXlkqS4uDiv7cuXL1fHjh3Vp08fzZ49W0eOHPHsy8vLU2pqqhISEjzb0tLSVFFRoZ07dzb6u1PZAwDMoYmG8YuLi2W1Wj2bj1fV/5jb7dbtt9+un//85+rTp49n+3XXXafk5GQlJiZq+/btmjVrlgoKCvTaa69JkpxOp1eil+T57HQ6Gx06yR4AYApN9bhcq9XqlewbIzMzU//+97+1adMmr+0333yz5+fU1FR17txZQ4cO1d69e3X22WeferA/wjA+AADNaMqUKVq7dq3ee+89denS5Sf7Dhw4UJJUWFgoSbLb7SotLfXq0/D5RPP8x0OyBwCYg9EEzZfLGYamTJmiVatW6d1339VZZ5110mPy8/MlSZ07d5YkORwO7dixQ2VlZZ4+OTk5slqtSklJaXQsDOMDAEyjJd9cl5mZqRUrVugf//iHYmNjPXPsNptNkZGR2rt3r1asWKFRo0apQ4cO2r59u6ZNm6bBgwerb9++kqThw4crJSVF48eP16JFi+R0OjVnzhxlZmY2aq1AAyp7AACawZIlS1ReXq4hQ4aoc+fOnvbiiy9KksLCwvTOO+9o+PDh6tWrl+644w6NHTtWa9as8ZwjODhYa9euVXBwsBwOh66//npNmDDB6778xqCyBwCYg2Ecbf4c71P3n+6flJSk3Nzck54nOTlZ69at8+naP0ayBwCYQlOtxm+LGMYHACDAUdkDAMzBxK+4JdkDAEzB4j7a/Dm+rWIYHwCAAEdlD/UZWKn/d+sBnZN6RB3s9Zr72zOV96bNs//6O5waMvqQOiXWqa7WosIdkXr2AbsKPvn+FY3dU49o4l371aPfEbldFm1aZ9MTcxNVfSS4Nb4S4PG3P9n1wkPeTxrrcna1nn5/jyRp3Qsd9N6q9ircEakjlcF6dfcOxdhcXv0r/husx+ecoQ9zbLIESRePOqTJC75WZHQbLvXMyMTD+FT2UESUW5/vjNBf/nj8xzh+/Xm4su86Q7/7RQ/dcVV3OYvDlPX3z2WLq5ckxSXU6YGVn6tkX7hu++U5umtcNyX3rNaMR4pb8msAJ5Tc8zv9Pf/fnvbQ6s88+6q/C9IFQyp0zdTSEx7/f1OS9WVBpLJW7tX85z7Xjg9j9MjMpJYIHU2oqd561xa1arLfuHGjrrjiCiUmJspisWj16tWtGY5pffyeVc8t6qzNP6jmf+i9Ve31yfuxchaF68v/ROivcxMVbXXrrJTvJEkDh1Wovt6iv/zxDH21N0L/+TRKi2d10SW/LFfimTUt+VWA4woOluLi6z3N1uH7yn3MpAP6zdQy9Rpw5LjHFn0Wro/fs2ran4vU6/wj6jOwSrfe95Vy/9FO3zoZHG1TGu6z96e1Ua2a7KuqqtSvXz9lZ2e3ZhjwQUioW6Ou/1aV5UH6fFekJCk03K36OosMw+LpV1t99Ffr3IuqWiVO4Ie+3hema887VxmDeuuBzK4q+yq00cfu/jhaMbZ69ej3nWfb+ZccliVI2vODqSzgdNaqf5aOHDlSI0eObHT/mpoa1dR8XylWVFQ0R1g4joHDKjR7yZcKj3TrYGmIZl9ztioOHv31+XRTrH53b4munlym1U91VESUW7/9435JUlx8XWuGDajX+VWa8ch36nJ2jQ6WheqFP9t1x6/O0RPv7VFUzMnn3A8eCFG7DvVe24JDpNh29TpYRmXflvBQnTYiKytLNpvN05KSmDNrKfkfROvWy3to2pXd9fEGq+564kvZOhxN5F/+J0J/ur2rxv7ugP65d4f+nr9LzuIwHSwL8ar2gdZw4S8Oa/AV5eqWUq0LhhzWfS98rsqKYG38Z7vWDg0trYXfenc6aVPJfvbs2SovL/e04mIWgLWUmu+CVfJFuPZsi9bDdyTJVS+NuPagZ/97q9rr2v7n6rrzU/T/zj1Xf/tTgmwd6rX/y7BWjBo4VozNpS7dalTyRePeGBbXqV6HvvWu4F310uFDIYqLrz/BUcDppU0l+/DwcFmtVq+G1mEJkkLDj/0z99A3oao+EqxLRx9SXU2Qtm2MbYXogBP7ripIJV+GNXqKqfcFVaosD9Fn2yM92/I3xcpwS73OY01KW2Lm1fhMOEERUS4lnlXr+WxPqlW3c7/T4UPBqjgYrOtuK1Pe21YdLA2VNa5eV974jTra6/T+mnaeY6688Rvt+jhK31UF6/zBh3XT3SV6ZmFnVVVwnz1a11/nJWrQ8HLFd6nTt84Q/e1PnRUcJA351X8lSQfLQvTfslCV7Ds6CrVvT4Siot3qdEatrO1d6npOjS64rEKPzEjS1P/7Sq46i7LnnKFLRx9SBzuVfZvSwm+9O52Q7KEe/b7Tg6/u9Xy+ZV6JJOntF9tr8Z1d1KV7je7+f1/IGufS4f8G6z+fRumOX3XXl/+J8BzTs/8Rjb/DqYhot74qDNfiP3TR+lfjWvy7AD/2zf5QZd16pg7/N1i2DvU698IqPbL2P2r3v9vvXn++o9dDd2b86hxJ0h0PF2n4b45OVc36y5fKvquL7vz12Z6H6tx639ct/2WAU2QxTvbC3WZUWVmpwsJCSdJ5552nhx56SJdddpni4uLUtWvXkx5fUVEhm82mIRqtEEvjb6UB2pK3SvJbOwSg2VQcdqt9j89VXl7ebFOzDbnCMXK+QkIjTn7ACdTXVSvvjXuaNdbm0qqV/ccff6zLLrvM83n69OmSpIyMDC1btqyVogIABCQTPy63VZP9kCFD1IoDCwAAmAJz9gAAUzDzQ3VI9gAAc3AbR5s/x7dRJHsAgDmYeM6+TT1UBwAA+I7KHgBgChb5OWffZJG0PJI9AMAcTPwEPYbxAQAIcFT2AABT4NY7AAACHavxAQBAoKKyBwCYgsUwZPFjkZ0/x7Y2kj0AwBzc/2v+HN9GMYwPAECAo7IHAJgCw/gAAAQ6E6/GJ9kDAMyBJ+gBAIBARWUPADAFnqAHAECgYxgfAAAEKpI9AMAULG7/my+ysrJ04YUXKjY2VvHx8brqqqtUUFDg1ae6ulqZmZnq0KGDYmJiNHbsWJWWlnr1KSoqUnp6uqKiohQfH6+ZM2eqvr7ep1hI9gAAc2gYxven+SA3N1eZmZnasmWLcnJyVFdXp+HDh6uqqsrTZ9q0aVqzZo1efvll5ebmqqSkRGPGjPHsd7lcSk9PV21trTZv3qznnntOy5Yt0z333ONTLBbDaLuTEBUVFbLZbBqi0QqxhLZ2OECzeKskv7VDAJpNxWG32vf4XOXl5bJarc1zjYZccdFdCgmJOOXz1NdXa8O/7j/lWA8cOKD4+Hjl5uZq8ODBKi8vV6dOnbRixQpdffXVkqQ9e/aod+/eysvL06BBg/TGG2/ol7/8pUpKSpSQkCBJWrp0qWbNmqUDBw4oLCysUdemsgcAmIPRBE1H/3j4YaupqWnU5cvLyyVJcXFxkqStW7eqrq5Ow4YN8/Tp1auXunbtqry8PElSXl6eUlNTPYlektLS0lRRUaGdO3c2+quT7AEAptDwuFx/miQlJSXJZrN5WlZW1kmv7Xa7dfvtt+vnP/+5+vTpI0lyOp0KCwtTu3btvPomJCTI6XR6+vww0Tfsb9jXWNx6BwCAD4qLi72G8cPDw096TGZmpv79739r06ZNzRnaCZHsAQDm0ET32VutVp/m7KdMmaK1a9dq48aN6tKli2e73W5XbW2tDh065FXdl5aWym63e/r861//8jpfw2r9hj6NwTA+AMAcDH3/TvtTaT7+nWAYhqZMmaJVq1bp3Xff1VlnneW1f8CAAQoNDdX69es92woKClRUVCSHwyFJcjgc2rFjh8rKyjx9cnJyZLValZKS0uhYqOwBAKbQ0q+4zczM1IoVK/SPf/xDsbGxnjl2m82myMhI2Ww2TZw4UdOnT1dcXJysVqumTp0qh8OhQYMGSZKGDx+ulJQUjR8/XosWLZLT6dScOXOUmZnZqOmDBiR7AACawZIlSyRJQ4YM8dr+7LPP6oYbbpAkPfzwwwoKCtLYsWNVU1OjtLQ0Pf74456+wcHBWrt2rSZPniyHw6Ho6GhlZGRo/vz5PsVCsgcAmIMhP+fsfezeiGtFREQoOztb2dnZJ+yTnJysdevW+XbxHyHZAwDMgRfhAACAQEVlDwAwB7cki5/Ht1EkewCAKbT0avzTCcP4AAAEOCp7AIA5mHiBHskeAGAOJk72DOMDABDgqOwBAOZg4sqeZA8AMAduvQMAILBx6x0AAAhYVPYAAHNgzh4AgADnNiSLHwnb3XaTPcP4AAAEOCp7AIA5MIwPAECg8zPZq+0me4bxAQAIcFT2AABzYBgfAIAA5zbk11A8q/EBAMDpisoeAGAOhvto8+f4NopkDwAwB+bsAQAIcMzZAwCAQEVlDwAwB4bxAQAIcIb8TPZNFkmLYxgfAIAAR2UPADAHhvEBAAhwbrckP+6Vd7fd++wZxgcAIMBR2QMAzIFhfAAAApyJkz3D+AAABDgqewCAOZj4cbkkewCAKRiGW4Yfb67z59jWRrIHAJiDYfhXnTNnDwAATldU9gAAczD8nLOnsgcA4DTndvvffLBx40ZdccUVSkxMlMVi0erVq73233DDDbJYLF5txIgRXn0OHjyocePGyWq1ql27dpo4caIqKyt9/uokewAAmkFVVZX69eun7OzsE/YZMWKE9u/f72l///vfvfaPGzdOO3fuVE5OjtauXauNGzfq5ptv9jkWhvEBAObQwsP4I0eO1MiRI3+yT3h4uOx2+3H37d69W2+++aY++ugjXXDBBZKkxx57TKNGjdKf/vQnJSYmNjoWKnsAgCkYbrffTZIqKiq8Wk1NzSnHtGHDBsXHx6tnz56aPHmyvv32W8++vLw8tWvXzpPoJWnYsGEKCgrShx9+6NN1SPYAAPggKSlJNpvN07Kysk7pPCNGjNDzzz+v9evX6//+7/+Um5urkSNHyuVySZKcTqfi4+O9jgkJCVFcXJycTqdP12IYHwBgDk00jF9cXCyr1erZHB4efkqnu+aaazw/p6amqm/fvjr77LO1YcMGDR069NTjPA4qewCAObgN/5skq9Xq1U412f9Yt27d1LFjRxUWFkqS7Ha7ysrKvPrU19fr4MGDJ5znPxGSPQAAp4GvvvpK3377rTp37ixJcjgcOnTokLZu3erp8+6778rtdmvgwIE+nZthfACAORiGJD+eb+/javzKykpPlS5J+/btU35+vuLi4hQXF6d58+Zp7Nixstvt2rt3r/7whz+oe/fuSktLkyT17t1bI0aM0KRJk7R06VLV1dVpypQpuuaaa3xaiS9R2QMATMJwG343X3z88cc677zzdN5550mSpk+frvPOO0/33HOPgoODtX37dl155ZXq0aOHJk6cqAEDBuj999/3mhZYvny5evXqpaFDh2rUqFG6+OKL9de//tXn705lDwAwB8Mt/yp7344dMmSIjJ8YDXjrrbdOeo64uDitWLHCp+seD5U9AAABjsoeAGAKhtuQYTn1W+9+qko/3ZHsAQDm0MLD+KeTNp3sG/7KqledX89JAE5nFYfb7j8wwMlUVB79/W6JqtnfXFGvuqYLpoW16WR/+PBhSdImrWvlSIDm075Ha0cANL/Dhw/LZrM1y7nDwsJkt9u1yel/rrDb7QoLC2uCqFqWxWjDkxBut1slJSWKjY2VxWJp7XBMoaKiQklJScc8LhIIBPx+tzzDMHT48GElJiYqKKj51oxXV1ertrbW7/OEhYUpIiKiCSJqWW26sg8KClKXLl1aOwxTanhMJBCI+P1uWc1V0f9QREREm0zSTYVb7wAACHAkewAAAhzJHj4JDw/Xvffe22RveQJOJ/x+I1C16QV6AADg5KjsAQAIcCR7AAACHMkeAIAAR7IHACDAkezRaNnZ2TrzzDMVERGhgQMH6l//+ldrhwQ0iY0bN+qKK65QYmKiLBaLVq9e3dohAU2KZI9GefHFFzV9+nTde++92rZtm/r166e0tDSVlZW1dmiA36qqqtSvXz9lZ2e3dihAs+DWOzTKwIEDdeGFF+ovf/mLpKPvJUhKStLUqVN15513tnJ0QNOxWCxatWqVrrrqqtYOBWgyVPY4qdraWm3dulXDhg3zbAsKCtKwYcOUl5fXipEBABqDZI+T+uabb+RyuZSQkOC1PSEhQU6ns5WiAgA0FskeAIAAR7LHSXXs2FHBwcEqLS312l5aWiq73d5KUQEAGotkj5MKCwvTgAEDtH79es82t9ut9evXy+FwtGJkAIDGCGntANA2TJ8+XRkZGbrgggt00UUX6ZFHHlFVVZVuvPHG1g4N8FtlZaUKCws9n/ft26f8/HzFxcWpa9eurRgZ0DS49Q6N9pe//EUPPvignE6n+vfvr8WLF2vgwIGtHRbgtw0bNuiyyy47ZntGRoaWLVvW8gEBTYxkDwBAgGPOHgCAAEeyBwAgwJHsAQAIcCR7AAACHMkeAIAAR7IHACDAkewBAAhwJHsAAAIcyR7w0w033KCrrrrK83nIkCG6/fbbWzyODRs2yGKx6NChQyfsY7FYtHr16kafc+7cuerfv79fcX3xxReyWCzKz8/36zwATh3JHgHphhtukMVikcViUVhYmLp376758+ervr6+2a/92muvacGCBY3q25gEDQD+4kU4CFgjRozQs88+q5qaGq1bt06ZmZkKDQ3V7Nmzj+lbW1ursLCwJrluXFxck5wHAJoKlT0CVnh4uOx2u5KTkzV58mQNGzZM//znPyV9P/R+//33KzExUT179pQkFRcX69e//rXatWunuLg4jR49Wl988YXnnC6XS9OnT1e7du3UoUMH/eEPf9CPXy/x42H8mpoazZo1S0lJSQoPD1f37t319NNP64svvvC8fKV9+/ayWCy64YYbJB19hXBWVpbOOussRUZGql+/fnrllVe8rrNu3Tr16NFDkZGRuuyyy7zibKxZs2apR48eioqKUrdu3XT33Xerrq7umH5PPPGEkpKSFBUVpV//+tcqLy/32v/UU0+pd+/eioiIUK9evfT444/7HAuA5kOyh2lERkaqtrbW83n9+vUqKChQTk6O1q5dq7q6OqWlpSk2Nlbvv/++PvjgA8XExGjEiBGe4/785z9r2bJleuaZZ7Rp0yYdPHhQq1at+snrTpgwQX//+9+1ePFi7d69W0888YRiYmKUlJSkV199VZJUUFCg/fv369FHH5UkZWVl6fnnn9fSpUu1c+dOTZs2Tddff71yc3MlHf2jZMyYMbriiiuUn5+vm266SXfeeafP/09iY2O1bNky7dq1S48++qiefPJJPfzww159CgsL9dJLL2nNmjV688039cknn+jWW2/17F++fLnuuece3X///dq9e7cWLlyou+++W88995zP8QBoJgYQgDIyMozRo0cbhmEYbrfbyMnJMcLDw40ZM2Z49ickJBg1NTWeY/72t78ZPXv2NNxut2dbTU2NERkZabz11luGYRhG586djUWLFnn219XVGV26dPFcyzAM49JLLzVuu+02wzAMo6CgwJBk5OTkHDfO9957z5Bk/Pe///Vsq66uNqKioozNmzd79Z04caJx7bXXGoZhGLNnzzZSUlK89s+aNeuYc/2YJGPVqlUn3P/ggw8aAwYM8Hy+9957jeDgYOOrr77ybHvjjTeMoKAgY//+/YZhGMbZZ59trFixwus8CxYsMBwOh2EYhrFv3z5DkvHJJ5+c8LoAmhdz9ghYa9euVUxMjOrq6uR2u3Xddddp7ty5nv2pqale8/SffvqpCgsLFRsb63We6upq7d27V+Xl5dq/f78GDhzo2RcSEqILLrjgmKH8Bvn5+QoODtall17a6LgLCwt15MgRXX755V7ba2trdd5550mSdu/e7RWHJDkcjkZfo8GLL76oxYsXa+/evaqsrFR9fb2sVqtXn65du+qMM87wuo7b7VZBQYFiY2O1d+9eTZw4UZMmTfL0qa+vl81m8zkeAM2DZI+Addlll2nJkiUKCwtTYmKiQkK8f92jo6O9PldWVmrAgAFavnz5Mefq1KnTKcUQGRnp8zGVlZWSpNdff90ryUpH1yE0lby8PI0bN07z5s1TWlqabDabVq5cqT//+c8+x/rkk08e88dHcHBwk8UKwD8kewSs6Ohode/evdH9zz//fL344ouKj48/prpt0LlzZ3344YcaPHiwpKMV7NatW3X++ecft39qaqrcbrdyc3M1bNiwY/Y3jCy4XC7PtpSUFIWHh6uoqOiEIwK9e/f2LDZssGXLlpN/yR/YvHmzkpOTddddd3m2ffnll8f0KyoqUklJiRITEz3XCQoKUs+ePZWQkKDExER9/vnnGjdunE/XB9ByWKAH/M+4cePUsWNHjR49Wu+//7727dunDRs26Pe//72++uorSdJtt92mBx54QKtXr9aePXt06623/uQ98meeeaYyMjL029/+VqtXr/ac86WXXpIkJScny2KxaO3atTpw4IAqKysVGxurGTNmaNq0aXruuee0d+9ebdu2TY899phn0dstt9yizz77TDNnzlRBQYFWrFihZcuW+fR9zznnHBUVFWnlypXau3evFi9efNzFhhEREcrIyNCnn36q999/X7///e/161//Wna7XZI0b948ZWVlafHixfrPf/6jHTt26Nlnn9VDDz3kUzwAmg/JHvifqKgobdy4UV27dtWYMWPUu3dvTZw4UdXV1Z5K/4477tD48eOVkZEhh8Oh2NhY/epXv/rJ8y5ZskRXX321br31VvXq1UuTJk1SVVWVJOmMM87QvHnzdOeddyohIUFTpkyRJC1YsEB33323srKy1Lt3b40YMUKvv/66zjrrLElH59FfffVVrV69Wv369dPSpUu1cOFCn77vlVdeqWnTpmnKlCnq37+/Nm/erLvvvvuYft27d9eYMWM0atQoDR8+XH379vW6te6mm27SU089pWeffVapqam69NJLtWzZMk+sAFqfxTjRyiIAABAQqOwBAAhwJHsAAAIcyR4AgABHsgcAIMCR7AEACHAkewAAAhzJHgCAAEeyBwAgwJHsAQAIcCR7AAACHMkeAIAA9/8By801/ssj8NMAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "disp = ConfusionMatrixDisplay(confusion_matrix=cm)\n", "disp.plot()" ] }, { "cell_type": "markdown", "metadata": { "id": "xWWS51LpGv75" }, "source": [ "Now, let's look at the other metrics" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "JDQ4clUhG0E2", "outputId": "d4ecf921-195b-4110-c224-98df5b600840" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy: 0.7265625 \n", "Precision: 0.7434402332361516\n", "Recall: 0.785824345146379\n", "F1: 0.7640449438202247\n" ] } ], "source": [ "print('{:<10} {:<15}'.format('Accuracy:', accuracy_score(y_test, y_pred)))\n", "print('{:<10} {:<15}'.format('Precision:', precision_score(y_test, y_pred)))\n", "print('{:<10} {:<15}'.format('Recall:', recall_score(y_test, y_pred)))\n", "print('{:<10} {:<15}'.format('F1:', f1_score(y_test, y_pred)))" ] }, { "cell_type": "markdown", "metadata": { "id": "M3WtaK81L8iT" }, "source": [ "### Comparison to VEGA model" ] }, { "cell_type": "markdown", "metadata": { "id": "Tec9purNJ-OK" }, "source": [ "What do you think about the metrics? Is the kNN model performing good or bad?\n", "Let's compare it to the predictions of a kNN model trained on this same dataset and published as part of the [VEGA platform](https://www.vegahub.eu/portfolio-item/vega-qsar/). In the VEGA model, they have choosen k=4 with a similarity threshold of 0.7 (according to an internal similarity metric) Why is such threshold important?\n", "\n", "Also, the developers of the VEGA kNN model used the [leave-one-out](https://en.wikipedia.org/wiki/Cross-validation_(statistics)#Leave-one-out_cross-validation) approach to assess the performace of their model. Why is this approach specially suitable when using the kNN algorithm? " ] }, { "cell_type": "markdown", "metadata": { "id": "nyfyVxebUAAX" }, "source": [ "#### VEGA model" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 348 }, "id": "cD_XnbgGJ9PD", "outputId": "2b79c535-7ef7-4ae3-f575-75e48cbdb534" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy: 0.7999305314345259\n", "Precision: 0.8168631006346329\n", "Recall: 0.8319482917820868\n", "F1: 0.8243366880146386\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAGwCAYAAAA0bWYRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFhElEQVR4nO3de1hUdf4H8PdwmeE6A4gwjCCirCh5SytiWw1WA82fadq2pSbmLQs0Ma+VilpRmpqmaWVKbZi6W7re0lBTNFETRdOUQlEwGLQQRlAYmDm/P4jRWRiZYYaLnPfrec4Tc873nPMZH2I+8/lejkQQBAFEREQkWnZNHQARERE1LSYDREREIsdkgIiISOSYDBAREYkckwEiIiKRYzJAREQkckwGiIiIRM6hqQOwhl6vR15eHtzd3SGRSJo6HCIispAgCLh58yZUKhXs7Bru+2lZWRm0Wq3V15FKpXBycrJBRM3LfZ0M5OXlISAgoKnDICIiK+Xm5sLf379Brl1WVoagQDeor+msvpZSqUR2dnaLSwju62TA3d0dANDh03jYu8iaOBqihuE//nJTh0DUYCqFCqTe/trw97whaLVaqK/pcCW9HeTu9a8+aG7qEdjrMrRaLZOB5qS6a8DeRcZkgFosB4m0qUMganCN0dXr5i6Bm3v976NHy+2Ovq+TASIiInPpBD10VjyNRyfobRdMM8NkgIiIREEPAXrUPxuw5tzmjlMLiYiIRI6VASIiEgU99LCm0G/d2c0bkwEiIhIFnSBAJ9S/1G/Nuc0duwmIiIhEjpUBIiISBQ4gNI3JABERiYIeAnRMBmrFbgIiIiKRY2WAiIhEgd0EpjEZICIiUeBsAtPYTUBERCRyrAwQEZEo6P/crDm/pWIyQEREoqCzcjaBNec2d0wGiIhIFHQCrHxqoe1iaW44ZoCIiEjkWBkgIiJR4JgB05gMEBGRKOghgQ4Sq85vqdhNQEREJHKsDBARkSjoharNmvNbKiYDREQkCjoruwmsObe5YzcBERGRyLEyQEREosDKgGlMBoiISBT0ggR6wYrZBFac29yxm4CIiEjkWBkgIiJRYDeBaUwGiIhIFHSwg86KgrjOhrE0N0wGiIhIFAQrxwwIHDNARERElkhMTMTDDz8Md3d3+Pj4YMiQIcjMzDRqExERAYlEYrRNnDjRqE1OTg4GDhwIFxcX+Pj4YPr06aisrDRqc+DAAfTs2RMymQzBwcFISkqyKFYmA0REJArVYwas2Sxx8OBBxMbG4ujRo0hJSUFFRQWioqJQWlpq1G78+PHIz883bIsWLboTs06HgQMHQqvV4siRI/j888+RlJSEuXPnGtpkZ2dj4MCBiIyMREZGBqZMmYJx48Zhz549ZsfKbgIiIhIFnWAHnWDFmAELlyPevXu30eukpCT4+PggPT0dffr0Mex3cXGBUqms9Rrfffcdfv75Z+zduxe+vr7o0aMHFi5ciJkzZyIhIQFSqRRr1qxBUFAQlixZAgDo3LkzDh8+jGXLliE6OtqsWFkZICIisoBGozHaysvLzTqvuLgYAODl5WW0Pzk5Gd7e3ujSpQtmz56NW7duGY6lpaWha9eu8PX1NeyLjo6GRqPBuXPnDG369etndM3o6GikpaWZ/Z5YGSAiIlHQQwK9Fd+B9agqDQQEBBjtnzdvHhISEu59rl6PKVOm4LHHHkOXLl0M+4cPH47AwECoVCqcOXMGM2fORGZmJr755hsAgFqtNkoEABheq9Xqe7bRaDS4ffs2nJ2d63xvTAaIiEgUbLXOQG5uLuRyuWG/TCar89zY2FicPXsWhw8fNto/YcIEw89du3aFn58f+vbti4sXL6JDhw71jtVS7CYgIiKygFwuN9rqSgbi4uKwY8cOfP/99/D3979n27CwMABAVlYWAECpVKKgoMCoTfXr6nEGptrI5XKzqgIAkwEiIhKJ6gGE1myWEAQBcXFx2LJlC/bv34+goKA6z8nIyAAA+Pn5AQDCw8Px008/4dq1a4Y2KSkpkMvlCA0NNbTZt2+f0XVSUlIQHh5udqxMBoiISBSqxgxYt1kiNjYWX375JTZs2AB3d3eo1Wqo1Wrcvn0bAHDx4kUsXLgQ6enpuHz5MrZt24ZRo0ahT58+6NatGwAgKioKoaGheOGFF3D69Gns2bMHb775JmJjYw0ViYkTJ+LSpUuYMWMGLly4gI8++gibN29GfHy82bEyGSAiImoAq1evRnFxMSIiIuDn52fYNm3aBACQSqXYu3cvoqKi0KlTJ7z22msYNmwYtm/fbriGvb09duzYAXt7e4SHh2PkyJEYNWoUFixYYGgTFBSEnTt3IiUlBd27d8eSJUuwdu1as6cVAhxASEREIqG38tkE1bMJzCUI924fEBCAgwcP1nmdwMBA7Nq1655tIiIicOrUKYviuxuTASIiEgXrFx2ycNWh+wiTASIiEgU97GyyzkBLxDEDREREIsfKABERiYJOkEBnxWOIrTm3uWMyQEREoqCzcgChjt0ERERE1FKxMkBERKKgF+ygt2I2gZ6zCYiIiO5v7CYwjd0EREREIsfKABERiYIe1s0I0NsulGaHyQAREYmC9YsOtdxiest9Z0RERGQWVgaIiEgUrH82Qcv9/sxkgIiIREEPCfSwZswAVyAkIiK6r7EyYFrLfWdERERkFlYGiIhIFKxfdKjlfn9mMkBERKKgFyTQW7POQAt+amHLTXOIiIjILKwMEBGRKOit7CZoyYsOMRkgIiJRsP6phS03GWi574yIiIjMwsoAERGJgg4S6KxYOMiac5s7JgNERCQK7CYwreW+MyIiIjILKwNERCQKOlhX6tfZLpRmh8kAERGJArsJTGMyQEREosAHFZnWct8ZERERmYWVASIiEgUBEuitGDMgcGohERHR/Y3dBKa13HdGREREZmEyQEREolD9CGNrNkskJibi4Ycfhru7O3x8fDBkyBBkZmYajhcWFmLSpEkICQmBs7Mz2rZti8mTJ6O4uNjoOhKJpMa2ceNGozYHDhxAz549IZPJEBwcjKSkJItiZTJARESioPvzqYXWbJY4ePAgYmNjcfToUaSkpKCiogJRUVEoLS0FAOTl5SEvLw/vv/8+zp49i6SkJOzevRtjx46tca3169cjPz/fsA0ZMsRwLDs7GwMHDkRkZCQyMjIwZcoUjBs3Dnv27DE7Vo4ZICIiagC7d+82ep2UlAQfHx+kp6ejT58+6NKlC77++mvD8Q4dOuDtt9/GyJEjUVlZCQeHOx/RHh4eUCqVtd5nzZo1CAoKwpIlSwAAnTt3xuHDh7Fs2TJER0ebFSsrA0REJAq26ibQaDRGW3l5uVn3ry7/e3l53bONXC43SgQAIDY2Ft7e3njkkUewbt06CIJgOJaWloZ+/foZtY+OjkZaWppZcQFMBoiISCT0sLN6A4CAgAAoFArDlpiYWPe99XpMmTIFjz32GLp06VJrm99//x0LFy7EhAkTjPYvWLAAmzdvRkpKCoYNG4ZXXnkFH374oeG4Wq2Gr6+v0Tm+vr7QaDS4ffu2Wf827CYgIiKyQG5uLuRyueG1TCar85zY2FicPXsWhw8frvW4RqPBwIEDERoaioSEBKNjc+bMMfz84IMPorS0FIsXL8bkyZPr9wZqwcoAERGJgk6QWL0BgFwuN9rqSgbi4uKwY8cOfP/99/D3969x/ObNm+jfvz/c3d2xZcsWODo63vN6YWFhuHr1qqF7QqlUoqCgwKhNQUEB5HI5nJ2dzfq3YWWAiIhEoT7TA//3fEsIgoBJkyZhy5YtOHDgAIKCgmq00Wg0iI6Ohkwmw7Zt2+Dk5FTndTMyMuDp6WlIQsLDw7Fr1y6jNikpKQgPDzc7ViYDREQkCoKVTy0ULDw3NjYWGzZswH//+1+4u7tDrVYDABQKBZydnaHRaBAVFYVbt27hyy+/NAxIBIDWrVvD3t4e27dvR0FBAR599FE4OTkhJSUF77zzDqZNm2a4z8SJE7Fy5UrMmDEDY8aMwf79+7F582bs3LnT7FiZDBARETWA1atXAwAiIiKM9q9fvx6jR4/GyZMncezYMQBAcHCwUZvs7Gy0a9cOjo6OWLVqFeLj4yEIAoKDg7F06VKMHz/e0DYoKAg7d+5EfHw8li9fDn9/f6xdu9bsaYUAkwEiIhIJHSTQWfGwIUvPvXv6X20iIiLqbNO/f3/079+/zntFRETg1KlTFsV3NyYDREQkCnrB8n7//z2/peJsAiIiIpFjZUBkpOdK4fbf3yG9WAb7G5X4Y2YAysLuzJe1K6qE/F8FcMoogaRUB22oK4rGKaFT3TV1RquHIkkNl8MaoFJAeQ9XFE1QQe9R9etkd7MSnst+g+OVMtjd1EGnsEfZI3JoRvhAcLFv7LdMhFa+5RgzIwcP9SmCzFmHvCtOWDYzGL+edfuzhYAXXs1F/39eg6u8Ej+ny7FybhDyrhhPy3o44gaGx11FUKdSaMvt8NNxORa+3Knx3xDVi97KAYTWnNvcMRkQGUm5HhXtnHDr755otSjX+KAgoNW7ORAcJPhjVlsILnZw2/YHvBOu4NqKYAhOVf8jKNar4ZRegj+m+0NwsYfHp/nwei8Hvye2r7qMRIKyR9yhGe4DvdweDmotFJ/mw6NEhxvxNefYEjUkN3kllmw6h9NH5ZgzthOKCx3Rpl0ZSjR3/vz9Y0IenopRY8mMYKhzZRgVn4u31p/HS/17oEJb9Xv/WPQfePXti0ha0han04Jh7yAgsOOtpnpbVA96SKC3YsyANec2d80izVm1ahXatWsHJycnhIWF4fjx400dUotV3tMdN4f7ouxReY1jDvlaSH+5jaIJfqj4izMq28hQ9JIfJFo9nA9VraktKdXBdV8Rikf7QtvVDRUdnHEjrg1kmbfhmFn1h1Fws0dpfy9UBDtD5yNFeTc3lPb3gvTn0kZ9r0QA8I+XfsP1fCmWzQrGL2fcUXDVCScPeyA/p3o+t4Aho/OxcZU/ju71wuVMV7w/LRitfLX46xOFAAA7ewET51zG2vcCsesrJX677IycLBcc2uXddG+MyIaaPBnYtGkTpk6dinnz5uHkyZPo3r07oqOjce3ataYOTXwqqkbHCNK7fi3sJBAcJZBeqPqgd7x0G5JKAeXd3QxNKv1lqPR2hPSX2r8l2RVWwPmoBtoHXBsudiITHu17A7+edcPrH2biq2M/YuW20+j/zzurtSkDyuHlU4FTRxSGfbdKHJB52g2dHrwJAAh+oATeSi0EvQQrt51G8pETWPDZeQT+hZWB+4mtViBsiZo8GaieL/niiy8iNDQUa9asgYuLC9atW9fUoYlOZZuqD3X5lwWQlOiACj3cvrkOhz8qYX+jAgBgf6MSgoMEgqtx37/ewwH2NyqN9nkuzYXfcz/Db9wv0Dvb48YrqkZ7L0TVlAFlGDhcjd8uO+PNF0OxM1mJiXOy0e/pqi8cnt5Vv9s3fjdeAvbG71J4tq465hdQtezriMm5+GqVP+aN74SSYnu8l3wOboqKRnw3ZI3qMQPWbC1Vk74zrVaL9PR0o0cv2tnZoV+/frU+erG8vLzGoyPJhhwkKJwZAIc8LVSjLkD1/HnIzt5CWU83QGJ5Rlz8ohLX3++AP2YFwKFAC8V6dQMETXRvEgmQdc4Vny9pi4s/u+LbTb7YvckXTw4vqPvk6mvYVVXNNn3kjx/2tELWOTcsm1W1SEzvAYUNEjdRY2rSZOD333+HTqer9dGL1cs23i0xMdHosZEBAQGNFapoVHRwxvWlHZD3r05QfxaCP+YGwu6mDpW+Vd+adJ4OkFQKkJTqjM6zK6qEztN4PKre0xGV/jKUPSJH0UQ/uO25AbtCfouixlV43RE5WS5G+3IvOqO1X9W3/eqKQHWFoJqntxY3rjv+eQ0pACAn687sggqtHfJznOCjMu9Z9tT09JAYnk9Qr40DCJuH2bNno7i42LDl5ubWfRLVi+BqD73CAfZ55XC8eBtlj1QNOKxo7wzBQQLZmTuDAR1+K4fD7xXQdnQxdTlAX/UfSWULXrWDmqWf093hH2T8TPc2QWW4llc1XVadK0PhNUf0+Gux4biLWyVCupfgwil3AEDWWVdoyyVo0/7Odewd9PD1L8e13+p+fC01D8KfswnquwktOBlo0qmF3t7esLe3r/XRi0qlskZ7mUxm1nOjyTTJbR0c1FrDa/trWjhm34bezR661lI4HSmGXu4AnbcjHHPKoPhMjbJH3FHeo2rAoOBqj9K+HlCsV0PvZg/BxQ6KtfkoD3FGRUhVMiBLvwn7okpog50hONvBIaccii8KUN7JBTofaZO8bxKvretVWLL5LP758lWk7mqFkG4lGPDPAqx4s/2fLSTYmuSH5165it8uO6EgV4YX4nPxR4EUR1K8AFQNKNy1wRcvvHoVv+fLUPCbDM+MzwMAHPq2VRO9M7JUYz+18H7SpMmAVCpFr169sG/fPgwZMgQAoNfrsW/fPsTFxTVlaC2W48UytJ572fDaY31VIlYa6YGiSW1gf6MSivVq2BfroPNwwK0IBW7+o7XRNYpfVAISNVotzgUq9Cjv4YaiCX6G44LUDi57b0CxXg1JpQBdK0fcflSOm0M5DYsa3y8/uWHhKyEYPe0KhsddhTrXCR+/3Q7fb7vze/3vT1RwctZh8luX4CavxLkTcswZ09mwxgAArH0vEDqdBNPez4LMSY8LGW6Y9UKo0XoFRPcriVDXUxIa2KZNmxATE4OPP/4YjzzyCD744ANs3rwZFy5cqDGW4H9pNBooFAp0TJ4FexdWDKhlChiZ3dQhEDWYSkGL/bc2ori4GHJ5zfVPbKH6s+LplBfh6Fr/6mRFqRZbnljfoLE2lSZPaf/5z3/i+vXrmDt3LtRqNXr06IHdu3fXmQgQERFZgt0EpjV5MgAAcXFx7BYgIiJqIs0iGSAiImpofDaBaUwGiIhIFNhNYNp9tc4AERER2R4rA0REJAqsDJjGZICIiESByYBp7CYgIiISOVYGiIhIFFgZMI3JABERiYIA66YHtuTHrDEZICIiUWBlwDSOGSAiIhI5VgaIiEgUWBkwjckAERGJApMB09hNQEREJHKsDBARkSiwMmAakwEiIhIFQZBAsOID3Zpzmzt2ExAREYkcKwNERCQKekisWnTImnObO1YGiIhIFKrHDFizWSIxMREPP/ww3N3d4ePjgyFDhiAzM9OoTVlZGWJjY9GqVSu4ublh2LBhKCgoMGqTk5ODgQMHwsXFBT4+Ppg+fToqKyuN2hw4cAA9e/aETCZDcHAwkpKSLIqVyQAREVEDOHjwIGJjY3H06FGkpKSgoqICUVFRKC0tNbSJj4/H9u3b8e9//xsHDx5EXl4ehg4dajiu0+kwcOBAaLVaHDlyBJ9//jmSkpIwd+5cQ5vs7GwMHDgQkZGRyMjIwJQpUzBu3Djs2bPH7FglgiDct8stazQaKBQKdEyeBXsXWVOHQ9QgAkZmN3UIRA2mUtBi/62NKC4uhlwub5B7VH9WPLLlVTi41v+zorK0HMefXl7vWK9fvw4fHx8cPHgQffr0QXFxMVq3bo0NGzbgmWeeAQBcuHABnTt3RlpaGh599FF8++23+L//+z/k5eXB19cXALBmzRrMnDkT169fh1QqxcyZM7Fz506cPXvWcK/nnnsORUVF2L17t1mxsTJARESiYKtuAo1GY7SVl5ebdf/i4mIAgJeXFwAgPT0dFRUV6Nevn6FNp06d0LZtW6SlpQEA0tLS0LVrV0MiAADR0dHQaDQ4d+6coc3d16huU30NczAZICIiUaieWmjNBgABAQFQKBSGLTExsc576/V6TJkyBY899hi6dOkCAFCr1ZBKpfDw8DBq6+vrC7VabWhzdyJQfbz62L3aaDQa3L5926x/G84mICIiskBubq5RN4FMVnfXQ2xsLM6ePYvDhw83ZGj1xmSAiIhEQbByBcLqyoBcLrdozEBcXBx27NiB1NRU+Pv7G/YrlUpotVoUFRUZVQcKCgqgVCoNbY4fP250verZBne3+d8ZCAUFBZDL5XB2djYrRnYTEBGRKAgABMGKzdL7CQLi4uKwZcsW7N+/H0FBQUbHe/XqBUdHR+zbt8+wLzMzEzk5OQgPDwcAhIeH46effsK1a9cMbVJSUiCXyxEaGmpoc/c1qttUX8McrAwQERE1gNjYWGzYsAH//e9/4e7ubujjVygUcHZ2hkKhwNixYzF16lR4eXlBLpdj0qRJCA8Px6OPPgoAiIqKQmhoKF544QUsWrQIarUab775JmJjYw3dExMnTsTKlSsxY8YMjBkzBvv378fmzZuxc+dOs2NlMkBERKKghwSSRlyBcPXq1QCAiIgIo/3r16/H6NGjAQDLli2DnZ0dhg0bhvLyckRHR+Ojjz4ytLW3t8eOHTvw8ssvIzw8HK6uroiJicGCBQsMbYKCgrBz507Ex8dj+fLl8Pf3x9q1axEdHW12rEwGiIhIFBr7QUXmLOPj5OSEVatWYdWqVSbbBAYGYteuXfe8TkREBE6dOmVRfHfjmAEiIiKRY2WAiIhEQS9IILGiMmDNTITmjskAERGJQvWsAGvOb6nYTUBERCRyrAwQEZEoNPYAwvsJkwEiIhIFJgOmMRkgIiJR4ABC0zhmgIiISORYGSAiIlHgbALTmAwQEZEoVCUD1owZsGEwzQy7CYiIiESOlQEiIhIFziYwjckAERGJgvDnZs35LRW7CYiIiESOlQEiIhIFdhOYxmSAiIjEgf0EJjEZICIicbCyMoAWXBngmAEiIiKRY2WAiIhEgSsQmsZkgIiIRIEDCE1jNwEREZHIsTJARETiIEisGwTYgisDTAaIiEgUOGbANHYTEBERiRwrA0REJA5cdMgkJgNERCQKnE1gmlnJwLZt28y+4FNPPVXvYIiIiKjxmZUMDBkyxKyLSSQS6HQ6a+IhIiJqOC241G8Ns5IBvV7f0HEQERE1KHYTmGbVbIKysjJbxUFERNSwBBtsLZTFyYBOp8PChQvRpk0buLm54dKlSwCAOXPm4LPPPrN5gERERNSwLE4G3n77bSQlJWHRokWQSqWG/V26dMHatWttGhwREZHtSGywtUwWJwNffPEFPvnkE4wYMQL29vaG/d27d8eFCxdsGhwREZHNNHI3QWpqKgYNGgSVSgWJRIKtW7caHZdIJLVuixcvNrRp165djePvvvuu0XXOnDmD3r17w8nJCQEBAVi0aJFlgaIeycBvv/2G4ODgGvv1ej0qKiosDoCIiKglKi0tRffu3bFq1apaj+fn5xtt69atg0QiwbBhw4zaLViwwKjdpEmTDMc0Gg2ioqIQGBiI9PR0LF68GAkJCfjkk08sitXiRYdCQ0Nx6NAhBAYGGu3/z3/+gwcffNDSyxERETUOG61AqNFojHbLZDLIZLIazQcMGIABAwaYvJxSqTR6/d///heRkZFo37690X53d/cabaslJydDq9Vi3bp1kEqleOCBB5CRkYGlS5diwoQJ5rwrAPWoDMydOxdxcXF47733oNfr8c0332D8+PF4++23MXfuXEsvR0RE1Diqn1pozQYgICAACoXCsCUmJlodWkFBAXbu3ImxY8fWOPbuu++iVatWePDBB7F48WJUVlYajqWlpaFPnz5GY/iio6ORmZmJGzdumH1/iysDgwcPxvbt27FgwQK4urpi7ty56NmzJ7Zv344nnnjC0ssRERHdV3JzcyGXyw2va6sKWOrzzz+Hu7s7hg4darR/8uTJ6NmzJ7y8vHDkyBHMnj0b+fn5WLp0KQBArVYjKCjI6BxfX1/DMU9PT7PuX69nE/Tu3RspKSn1OZWIiKhJ2OoRxnK53CgZsIV169ZhxIgRcHJyMto/depUw8/dunWDVCrFSy+9hMTERJskIdXq/aCiEydO4Pz58wCqxhH06tXLZkERERHZXDN9auGhQ4eQmZmJTZs21dk2LCwMlZWVuHz5MkJCQqBUKlFQUGDUpvq1qXEGtbE4Gbh69Sqef/55/PDDD/Dw8AAAFBUV4a9//Ss2btwIf39/Sy9JREQkWp999hl69eqF7t2719k2IyMDdnZ28PHxAQCEh4fjjTfeQEVFBRwdHQEAKSkpCAkJMbuLAKjHAMJx48ahoqIC58+fR2FhIQoLC3H+/Hno9XqMGzfO0ssRERE1DhsNIDRXSUkJMjIykJGRAQDIzs5GRkYGcnJyDG00Gg3+/e9/1/r5mZaWhg8++ACnT5/GpUuXkJycjPj4eIwcOdLwQT98+HBIpVKMHTsW586dw6ZNm7B8+XKj7gVzWFwZOHjwII4cOYKQkBDDvpCQEHz44Yfo3bu3pZcjIiJqFBKharPmfEucOHECkZGRhtfVH9AxMTFISkoCAGzcuBGCIOD555+vcb5MJsPGjRuRkJCA8vJyBAUFIT4+3uiDXqFQ4LvvvkNsbCx69eoFb29vzJ0716JphUA9koGAgIBaFxfS6XRQqVSWXo6IiKhxNPKYgYiICAh1jFicMGGCyQ/unj174ujRo3Xep1u3bjh06JBlwf0Pi7sJFi9ejEmTJuHEiROGfSdOnMCrr76K999/36pgiIiIqPGZVRnw9PSERHKnr6S0tBRhYWFwcKg6vbKyEg4ODhgzZgyGDBnSIIESERFZpR79/jXOb6HMSgY++OCDBg6DiIiogTXTqYXNgVnJQExMTEPHQURERE2k3osOAUBZWRm0Wq3RPluvykRERGQTrAyYZPEAwtLSUsTFxcHHxweurq7w9PQ02oiIiJolwQZbC2VxMjBjxgzs378fq1evhkwmw9q1azF//nyoVCp88cUXDREjERERNSCLuwm2b9+OL774AhEREXjxxRfRu3dvBAcHIzAwEMnJyRgxYkRDxElERGQdziYwyeLKQGFhIdq3bw+ganxAYWEhAOBvf/sbUlNTbRsdERGRjVSvQGjN1lJZnAy0b98e2dnZAIBOnTph8+bNAKoqBtUPLiIiIqL7h8XJwIsvvojTp08DAGbNmoVVq1bByckJ8fHxmD59us0DJCIisgkOIDTJ4jED8fHxhp/79euHCxcuID09HcHBwejWrZtNgyMiIqKGZ9U6AwAQGBiIwMBAW8RCRETUYCSw8qmFNouk+TErGVixYoXZF5w8eXK9gyEiIqLGZ1YysGzZMrMuJpFImiQZUI04DweJY6Pfl6gxfJuX0dQhEDUYzU09PDs20s04tdAks5KB6tkDRERE9y0uR2ySxbMJiIiIqGWxegAhERHRfYGVAZOYDBARkShYu4ogVyAkIiKiFouVASIiEgd2E5hUr8rAoUOHMHLkSISHh+O3334DAPzrX//C4cOHbRocERGRzXA5YpMsTga+/vprREdHw9nZGadOnUJ5eTkAoLi4GO+8847NAyQiIqKGZXEy8NZbb2HNmjX49NNP4eh4Z6Gfxx57DCdPnrRpcERERLbCRxibZvGYgczMTPTp06fGfoVCgaKiIlvEREREZHtcgdAkiysDSqUSWVlZNfYfPnwY7du3t0lQRERENscxAyZZnAyMHz8er776Ko4dOwaJRIK8vDwkJydj2rRpePnllxsiRiIiImpAFncTzJo1C3q9Hn379sWtW7fQp08fyGQyTJs2DZMmTWqIGImIiKzGRYdMszgZkEgkeOONNzB9+nRkZWWhpKQEoaGhcHNza4j4iIiIbIPrDJhU70WHpFIpQkNDbRkLERERNQGLk4HIyEhIJKZHVO7fv9+qgIiIiBqEtdMDWRm4o0ePHkavKyoqkJGRgbNnzyImJsZWcREREdkWuwlMsng2wbJly4y2lStX4vDhw5gyZYrRIkRERERilpqaikGDBkGlUkEikWDr1q1Gx0ePHg2JRGK09e/f36hNYWEhRowYAblcDg8PD4wdOxYlJSVGbc6cOYPevXvDyckJAQEBWLRokcWx2uyphSNHjsS6detsdTkiIiLbauR1BkpLS9G9e3esWrXKZJv+/fsjPz/fsH311VdGx0eMGIFz584hJSUFO3bsQGpqKiZMmGA4rtFoEBUVhcDAQKSnp2Px4sVISEjAJ598YlGsNntqYVpaGpycnGx1OSIiIpuy1dRCjUZjtF8mk0Emk9VoP2DAAAwYMOCe15TJZFAqlbUeO3/+PHbv3o0ff/wRDz30EADgww8/xJNPPon3338fKpUKycnJ0Gq1WLduHaRSKR544AFkZGRg6dKlRklDXSxOBoYOHWr0WhAE5Ofn48SJE5gzZ46llyMiIrqvBAQEGL2eN28eEhIS6nWtAwcOwMfHB56envj73/+Ot956C61atQJQ9SXbw8PDkAgAQL9+/WBnZ4djx47h6aefRlpaGvr06QOpVGpoEx0djffeew83btyAp6enWXFYnAwoFAqj13Z2dggJCcGCBQsQFRVl6eWIiIjuK7m5uZDL5YbXtVUFzNG/f38MHToUQUFBuHjxIl5//XUMGDAAaWlpsLe3h1qtho+Pj9E5Dg4O8PLyglqtBgCo1WoEBQUZtfH19TUca5BkQKfT4cUXX0TXrl3NvgEREVGzYKPZBHK53CgZqK/nnnvO8HPXrl3RrVs3dOjQAQcOHEDfvn2tvr4lLBpAaG9vj6ioKD6dkIiI7jvN/RHG7du3h7e3t+FhgEqlEteuXTNqU1lZicLCQsM4A6VSiYKCAqM21a9NjUWojcWzCbp06YJLly5ZehoRERHdw9WrV/HHH3/Az88PABAeHo6ioiKkp6cb2uzfvx96vR5hYWGGNqmpqaioqDC0SUlJQUhIiEUVfIuTgbfeegvTpk3Djh07kJ+fD41GY7QRERE1W434+OKSkhJkZGQgIyMDAJCdnY2MjAzk5OSgpKQE06dPx9GjR3H58mXs27cPgwcPRnBwMKKjowEAnTt3Rv/+/TF+/HgcP34cP/zwA+Li4vDcc89BpVIBAIYPHw6pVIqxY8fi3Llz2LRpE5YvX46pU6daFKvZYwYWLFiA1157DU8++SQA4KmnnjJallgQBEgkEuh0OosCICIiahSNvALhiRMnEBkZaXhd/QEdExOD1atX48yZM/j8889RVFQElUqFqKgoLFy40GhAYnJyMuLi4tC3b1/Y2dlh2LBhWLFiheG4QqHAd999h9jYWPTq1Qve3t6YO3euRdMKAUAiCIJZb8/e3h75+fk4f/78Pds9/vjjFgVgDY1GA4VCgQgMhoOEqx9Sy7QnL6OpQyBqMJqbenh2vITi4mKbDMqr9R5/flYEz3wH9rL6r4ejKy9D1nuvN2isTcXsykB1ztCYH/ZERES2YqtFh1oii6YW3utphURERM0aH1RkkkXJQMeOHetMCAoLC60KiIiIiBqXRcnA/Pnza6xASEREdD9gN4FpFiUDzz33XI2lEYmIiO4L7CYwyex1BjhegIiIqGWyeDYBERHRfYmVAZPMTgb0en1DxkFERNSgOGbANIsfYUxERHRfYmXAJIufTUBEREQtCysDREQkDqwMmMRkgIiIRIFjBkxjNwEREZHIsTJARETiwG4Ck5gMEBGRKLCbwDR2ExAREYkcKwNERCQO7CYwickAERGJA5MBk9hNQEREJHKsDBARkShI/tysOb+lYjJARETiwG4Ck5gMEBGRKHBqoWkcM0BERCRyrAwQEZE4sJvAJCYDREQkHi34A90a7CYgIiISOVYGiIhIFDiA0DQmA0REJA4cM2ASuwmIiIhEjpUBIiISBXYTmMZkgIiIxIHdBCaxm4CIiKgBpKamYtCgQVCpVJBIJNi6davhWEVFBWbOnImuXbvC1dUVKpUKo0aNQl5entE12rVrB4lEYrS9++67Rm3OnDmD3r17w8nJCQEBAVi0aJHFsTIZICIiUajuJrBms0RpaSm6d++OVatW1Th269YtnDx5EnPmzMHJkyfxzTffIDMzE0899VSNtgsWLEB+fr5hmzRpkuGYRqNBVFQUAgMDkZ6ejsWLFyMhIQGffPKJRbGym4CIiMShkbsJBgwYgAEDBtR6TKFQICUlxWjfypUr8cgjjyAnJwdt27Y17Hd3d4dSqaz1OsnJydBqtVi3bh2kUikeeOABZGRkYOnSpZgwYYLZsbIyQERE4iDYYEPVt/G7t/LycpuEV1xcDIlEAg8PD6P97777Llq1aoUHH3wQixcvRmVlpeFYWloa+vTpA6lUatgXHR2NzMxM3Lhxw+x7MxkgIiKyQEBAABQKhWFLTEy0+pplZWWYOXMmnn/+ecjlcsP+yZMnY+PGjfj+++/x0ksv4Z133sGMGTMMx9VqNXx9fY2uVf1arVabfX92ExARkSjYamphbm6u0Qe2TCazKq6Kigo8++yzEAQBq1evNjo2depUw8/dunWDVCrFSy+9hMTERKvvezcmA0REJA42GjMgl8uNkgFrVCcCV65cwf79++u8blhYGCorK3H58mWEhIRAqVSioKDAqE31a1PjDGrDbgIiIqImUJ0I/Prrr9i7dy9atWpV5zkZGRmws7ODj48PACA8PBypqamoqKgwtElJSUFISAg8PT3NjoWVASIiEgWJIEAi1L80YOm5JSUlyMrKMrzOzs5GRkYGvLy84Ofnh2eeeQYnT57Ejh07oNPpDH38Xl5ekEqlSEtLw7FjxxAZGQl3d3ekpaUhPj4eI0eONHzQDx8+HPPnz8fYsWMxc+ZMnD17FsuXL8eyZcssipXJABERiUMjTy08ceIEIiMjDa+r+/9jYmKQkJCAbdu2AQB69OhhdN7333+PiIgIyGQybNy4EQkJCSgvL0dQUBDi4+ONxhEoFAp89913iI2NRa9eveDt7Y25c+daNK0QYDJARETUICIiIiDco5pwr2MA0LNnTxw9erTO+3Tr1g2HDh2yOL67MRkgIiJR4IOKTGMyQERE4sAHFZnE2QREREQix8oAERGJArsJTGMyQERE4sBuApOYDBARkSiwMmAaxwwQERGJHCsDREQkDuwmMInJABERiUZLLvVbg90EREREIsfKABERiYMgVG3WnN9CMRkgIiJR4GwC09hNQEREJHKsDBARkThwNoFJTAaIiEgUJPqqzZrzWyp2ExAREYkcKwMiN/I1NV54rcBoX26WDOP6dPqflgLe+jIbD//9JhLGtEPaboXR0SeeLcTQCdfh374ct0rskbpDgVWv+zdw9EQ1bfzQBz/s8kBulgxSJz1CH7qFsW/kISC4HACgzpUiJiy01nPf+DgbfQYVAwCuXXXEh7P9cfoHdzi56vDEP25gzOt5sP/zr+bZY6747G0/5F50QvltO/i00WLgC39g6ITrjfI+qR7YTWASkwHC5QtOmPXP9obXOp2kRpunx/9uclbN0AnXMeyla1j7lgoXTrrAyUUP3wBtQ4VLdE9n0twwaPTv6NjjFnSVQNK7fnj9+Q749OAFOLno0VqlxVcZZ43O2fVlK/xntQ8e/vtNAIBOB8wZ1R6erSuxbNuvKLzmgMWTA2HvKGDM7HwAgJOLHk+9+DuCQsvg5KLHueOuWD7DH04uejw58o9Gf99UN84mMK1JuwlSU1MxaNAgqFQqSCQSbN26tSnDES2dDrhx3dGwaQqNc8T2D9zGsJeuY+nUgBrnuikqETMzH4tfbYvvt3gi/4oM2eedcfQ7RY22RI3hnQ2XEPXPQrQLKUOHB8rw2gc5uPabFL+ecQYA2NsDXj6VRtuRbxXoM6gIzq5VncInD7oj5xcnzFx5BR263MbDf7+JUTPysT3JGxXaqmQ5uOttRD5dhHYhZVAGaNF32A08FHETZ4+5Ntl7pzpUrzNgzdZCNWkyUFpaiu7du2PVqlVNGYbotQnSYsPJc0hKO4+ZK6+gdZs73+plznrMWnUFq95ogxvXHWuc27NPCewkgLeyAp8evIAvT/yMN9ZcRmsVKwPUPJRq7AEA7h66Wo//esYZF8+5IPr5O9/mfz7hinadyuDZutKw76GIm7h10x5XMp1qvU7WT874+YQruj5aYsPoiRpHk3YTDBgwAAMGDDC7fXl5OcrLyw2vNRpNQ4QlKhdOuuD9KQG4elEGL58KjHytAEu2ZOGlyBDcLrXHSwm/4ecTrkjbU/s3fWVgOSR2wHOTr2H1HBVKb9pj9Ew1EjdewsS+HVFZwTGq1HT0emDNvDZ44OEStOtUVmub3V+1Qtu/lOGBh28Z9t247gDP1hVG7Ty8KwzH7jaiVyiK/3CArlKCka+pMWBEoY3fBdkKuwlMu6/GDCQmJmL+/PlNHUaLcuJ7ueHn7PPOuHDKFf86/jP6PFWE4j8c0OOxErwS1dHk+XYSwFEq4KM5bXDyoDsAIPHlQHx1+hy6/7UE6QflJs8lamgrX/fHlQvOWLL111qPl9+W4Pstnhg+RV3veyzZkoXbpXY4f9IF695RQdWuHJFPF9X7etSAOIDQpPsqGZg9ezamTp1qeK3RaBAQULMfm+qvVGOPq5dkULXTIqhTGfzaafHNBePBVnM+vYyzx1wx45lgFF6r6jrI+UVmOF5c6ABNoQN82hh/syJqTCtfb4NjKXIs2ZKF1qrafxcP7fRA+W0J+v3D+Nu8Z+tKZJ4y7vsv+t3RcOxuyrZVXWJBnctQdN0RXy5RMhmg+859lQzIZDLIZLK6G1K9ObnooArUYt/XDkjd5oFvN3gZHf/k+1/wcYIKR7+r+sZ/7seqP5j+Hcrxe74UAODuUQm5VyUKfpM2bvBEqBrjteqNNjiyW4HF/8kyfFjXZs9XrfBolAYerYzHE4Q+VIqNK3xR9LsDPLyrPvxPprrDxV2Hth1r724AqrolKrTsGmuu2E1g2n2VDJDtjZ+bh6PfyXHtqhStlBV4YZoaOj1wYIsnigsdah00eO03KQpyq5Ky3y7JcGS3HC8vyMPyGf4ovWmHMa+rcTVLhtM/uDX22yHCytf98f0WTySsvwRnNz0Kr1X9mXN110HmfOev+W/ZUvx01BULv7xU4xo9H7+Jth3LsGhSW4x9Mw83rjsi6T0lBo3+HVJZ1TW2rfeGTxstAoKrkoOfjrrh6zU+GDyW6ww0W3xqoUlMBkTO268Csz+6AndPHYr/cMC5H10x5f/+guJC8381Fk9ui5fm52HBF9kQ9MCZo254Y0R76CprrldA1NB2fO4NAJg+7C9G+19bloOof97pDtizsRW8/SrQ6/GbNa5hbw8s+OISPpwVgPhBHeHkoke/fxQiZnq+oY2gB9Yl+kGdI4W9A6AKLMeYN/Iw8AWuMUD3H4kgNF2qU1JSgqysLADAgw8+iKVLlyIyMhJeXl5o27ZtnedrNBooFApEYDAcJDW/wRK1BHvyMpo6BKIGo7mph2fHSyguLoZc3jADjqs/K8IHLICDY+1TQ81RWVGGtG/nNmisTaVJKwMnTpxAZGSk4XX14MCYmBgkJSU1UVRERNQicTaBSU2aDERERKAJCxNEREQEjhkgIiKR4GwC05gMEBGROOiFqs2a81soJgNERCQOHDNgElfHICIiagB1PZlXEATMnTsXfn5+cHZ2Rr9+/fDrr8ZLZxcWFmLEiBGQy+Xw8PDA2LFjUVJi/DCsM2fOoHfv3nByckJAQAAWLVpkcaxMBoiISBQkuDNuoF6bhfer68m8ixYtwooVK7BmzRocO3YMrq6uiI6ORlnZnVUuR4wYgXPnziElJQU7duxAamoqJkyYYDiu0WgQFRWFwMBApKenY/HixUhISMAnn3xiUazsJiAiInFo5BUI7/VkXkEQ8MEHH+DNN9/E4MGDAQBffPEFfH19sXXrVjz33HM4f/48du/ejR9//BEPPfQQAODDDz/Ek08+iffffx8qlQrJycnQarVYt24dpFIpHnjgAWRkZGDp0qVGSUNdWBkgIiKygEajMdrKy8stvkZ2djbUajX69etn2KdQKBAWFoa0tDQAQFpaGjw8PAyJAAD069cPdnZ2OHbsmKFNnz59IJXeeRZMdHQ0MjMzcePGDbPjYTJARESiYFUXwV3TEgMCAqBQKAxbYmKixbGo1VWPzfb19TXa7+vrazimVqvh4+NjdNzBwQFeXl5GbWq7xt33MAe7CYiISBxsNJsgNzfXaDnilvA0XVYGiIiILCCXy422+iQDSqUSAFBQUGC0v6CgwHBMqVTi2rVrRscrKytRWFho1Ka2a9x9D3MwGSAiIlGQCILVm60EBQVBqVRi3759hn0ajQbHjh1DeHg4ACA8PBxFRUVIT083tNm/fz/0ej3CwsIMbVJTU1FRUWFok5KSgpCQEHh6epodD5MBIiISB70NNguUlJQgIyMDGRkZAKoGDWZkZCAnJwcSiQRTpkzBW2+9hW3btuGnn37CqFGjoFKpMGTIEABA586d0b9/f4wfPx7Hjx/HDz/8gLi4ODz33HNQqVQAgOHDh0MqlWLs2LE4d+4cNm3ahOXLlxse/GcujhkgIiJqAHU9mXfGjBkoLS3FhAkTUFRUhL/97W/YvXs3nJzuPGY5OTkZcXFx6Nu3L+zs7DBs2DCsWLHCcFyhUOC7775DbGwsevXqBW9vb8ydO9eiaYUAIBHu48cGVj+jOgKD4SBxbOpwiBrEnryMpg6BqMFoburh2fESiouLjQbl2fQef35W9Ok9Fw4OTnWfYEJlZRlSDy1o0FibCisDREQkDnw2gUlMBoiISBwaeQXC+wkHEBIREYkcKwNERCQKd68iWN/zWyomA0REJA7sJjCJ3QREREQix8oAERGJgkRftVlzfkvFZICIiMSB3QQmsZuAiIhI5FgZICIiceCiQyYxGSAiIlGw9smDtnxqYXPDbgIiIiKRY2WAiIjEgQMITWIyQERE4iAAsGZ6YMvNBZgMEBGROHDMgGkcM0BERCRyrAwQEZE4CLByzIDNIml2mAwQEZE4cAChSewmICIiEjlWBoiISBz0ACRWnt9CMRkgIiJR4GwC09hNQEREJHKsDBARkThwAKFJTAaIiEgcmAyYxG4CIiIikWNlgIiIxIGVAZOYDBARkThwaqFJTAaIiEgUOLXQNI4ZICIiEjlWBoiISBw4ZsAkJgNERCQOegGQWPGBrm+5yQC7CYiIiESOyQAREYlDdTeBNZsF2rVrB4lEUmOLjY0FAERERNQ4NnHiRKNr5OTkYODAgXBxcYGPjw+mT5+OyspKm/2TVGM3ARERiYSVYwZg2bk//vgjdDqd4fXZs2fxxBNP4B//+Idh3/jx47FgwQLDaxcXF8PPOp0OAwcOhFKpxJEjR5Cfn49Ro0bB0dER77zzjhXvoyYmA0RERA2gdevWRq/fffdddOjQAY8//rhhn4uLC5RKZa3nf/fdd/j555+xd+9e+Pr6okePHli4cCFmzpyJhIQESKVSm8XKbgIiIhIHG3UTaDQao628vLzOW2u1Wnz55ZcYM2YMJJI7Kx8lJyfD29sbXbp0wezZs3Hr1i3DsbS0NHTt2hW+vr6GfdHR0dBoNDh37pwN/2FYGSAiIrHQC7C01F/zfCAgIMBo97x585CQkHDPU7du3YqioiKMHj3asG/48OEIDAyESqXCmTNnMHPmTGRmZuKbb74BAKjVaqNEAIDhtVqtrv/7qAWTASIiIgvk5uZCLpcbXstksjrP+eyzzzBgwACoVCrDvgkTJhh+7tq1K/z8/NC3b19cvHgRHTp0sG3QdWAyQERE4iDoqzZrzgcgl8uNkoG6XLlyBXv37jV84zclLCwMAJCVlYUOHTpAqVTi+PHjRm0KCgoAwOQ4g/rimAEiIhKHRp5aWG39+vXw8fHBwIED79kuIyMDAODn5wcACA8Px08//YRr164Z2qSkpEAulyM0NLResZjCygAREYmDjcYMWHSKXo/169cjJiYGDg53PnIvXryIDRs24Mknn0SrVq1w5swZxMfHo0+fPujWrRsAICoqCqGhoXjhhRewaNEiqNVqvPnmm4iNjTWra8ISTAaIiIgayN69e5GTk4MxY8YY7ZdKpdi7dy8++OADlJaWIiAgAMOGDcObb75paGNvb48dO3bg5ZdfRnh4OFxdXRETE2O0LoGtMBkgIiJxaIIHFUVFRUGo5byAgAAcPHiwzvMDAwOxa9cui+9rKSYDREQkDgKsTAZsFkmzwwGEREREIsfKABERiUMTdBPcL5gMEBGROOj1AKxYZ0BvxbnNHLsJiIiIRI6VASIiEgd2E5jEZICIiMSByYBJ7CYgIiISOVYGiIhIHJpgOeL7BZMBIiISBUHQQ7DiqYXWnNvcMRkgIiJxEATrvt1zzAARERG1VKwMEBGROAhWjhlowZUBJgNERCQOej0gsaLfvwWPGWA3ARERkcixMkBEROLAbgKTmAwQEZEoCHo9BCu6CVry1EJ2ExAREYkcKwNERCQO7CYwickAERGJg14AJEwGasNuAiIiIpFjZYCIiMRBEABYs85Ay60MMBkgIiJREPQCBCu6CQQmA0RERPc5QQ/rKgOcWkhEREQtFCsDREQkCuwmMI3JABERiQO7CUy6r5OB6iytEhVWrSNB1JxpbrbcP0BEmpKq3+/G+NZt7WdFJSpsF0wzc18nAzdv3gQAHMauJo6EqOF4dmzqCIga3s2bN6FQKBrk2lKpFEqlEofV1n9WKJVKSKVSG0TVvEiE+7gTRK/XIy8vD+7u7pBIJE0djihoNBoEBAQgNzcXcrm8qcMhsin+fjc+QRBw8+ZNqFQq2Nk13Jj2srIyaLVaq68jlUrh5ORkg4ial/u6MmBnZwd/f/+mDkOU5HI5/1hSi8Xf78bVUBWBuzk5ObXID3Fb4dRCIiIikWMyQEREJHJMBsgiMpkM8+bNg0wma+pQiGyOv98kVvf1AEIiIiKyHisDREREIsdkgIiISOSYDBAREYkckwEiIiKRYzJAZlu1ahXatWsHJycnhIWF4fjx400dEpFNpKamYtCgQVCpVJBIJNi6dWtTh0TUqJgMkFk2bdqEqVOnYt68eTh58iS6d++O6OhoXLt2ralDI7JaaWkpunfvjlWrVjV1KERNglMLySxhYWF4+OGHsXLlSgBVz4UICAjApEmTMGvWrCaOjsh2JBIJtmzZgiFDhjR1KESNhpUBqpNWq0V6ejr69etn2GdnZ4d+/fohLS2tCSMjIiJbYDJAdfr999+h0+ng6+trtN/X1xdqtbqJoiIiIlthMkBERCRyTAaoTt7e3rC3t0dBQYHR/oKCAiiVyiaKioiIbIXJANVJKpWiV69e2Ldvn2GfXq/Hvn37EB4e3oSRERGRLTg0dQB0f5g6dSpiYmLw0EMP4ZFHHsEHH3yA0tJSvPjii00dGpHVSkpKkJWVZXidnZ2NjIwMeHl5oW3btk0YGVHj4NRCMtvKlSuxePFiqNVq9OjRAytWrEBYWFhTh0VktQMHDiAyMrLG/piYGCQlJTV+QESNjMkAERGRyHHMABERkcgxGSAiIhI5JgNEREQix2SAiIhI5JgMEBERiRyTASIiIpFjMkBERCRyTAaIiIhEjskAkZVGjx6NIUOGGF5HRERgypQpjR7HgQMHIJFIUFRUZLKNRCLB1q1bzb5mQkICevToYVVcly9fhkQiQUZGhlXXIaKGw2SAWqTRo0dDIpFAIpFAKpUiODgYCxYsQGVlZYPf+5tvvsHChQvNamvOBzgRUUPjg4qoxerfvz/Wr1+P8vJy7Nq1C7GxsXB0dMTs2bNrtNVqtZBKpTa5r5eXl02uQ0TUWFgZoBZLJpNBqVQiMDAQL7/8Mvr164dt27YBuFPaf/vtt6FSqRASEgIAyM3NxbPPPgsPDw94eXlh8ODBuHz5suGaOp0OU6dOhYeHB1q1aoUZM2bgfx/v8b/dBOXl5Zg5cyYCAgIgk8kQHByMzz77DJcvXzY8HMfT0xMSiQSjR48GUPWI6MTERAQFBcHZ2Rndu3fHf/7zH6P77Nq1Cx07doSzszMiIyON4jTXzJkz0bFjR7i4uKB9+/aYM2cOKioqarT7+OOPERAQABcXFzz77LMoLi42Or527Vp07twZTk5O6NSpEz766COLYyGipsNkgETD2dkZWq3W8Hrfvn3IzMxESkoKduzYgYqKCkRHR8Pd3R2HDh3CDz/8ADc3N/Tv399w3pIlS5CUlIR169bh8OHDKCwsxJYtW+5531GjRuGrr77CihUrcP78eXz88cdwc3NDQEAAvv76awBAZmYm8vPzsXz5cgBAYmIivvjiC6xZswbnzp1DfHw8Ro4ciYMHDwKoSlqGDh2KQYMGISMjA+PGjcOsWbMs/jdxd3dHUlISfv75Zyxfvhyffvopli1bZtQmKysLmzdvxvbt27F7926cOnUKr7zyiuF4cnIy5s6di7fffhvnz5/HO++8gzlz5uDzzz+3OB4iaiICUQsUExMjDB48WBAEQdDr9UJKSoogk8mEadOmGY77+voK5eXlhnP+9a9/CSEhIYJerzfsKy8vF5ydnYU9e/YIgiAIfn5+wqJFiwzHKyoqBH9/f8O9BEEQHn/8ceHVV18VBEEQMjMzBQBCSkpKrXF+//33AgDhxo0bhn1lZWWCi4uLcOTIEaO2Y8eOFZ5//nlBEARh9uzZQmhoqNHxmTNn1rjW/wIgbNmyxeTxxYsXC7169TK8njdvnmBvby9cvXrVsO/bb78V7OzshPz8fEEQBKFDhw7Chg0bjK6zcOFCITw8XBAEQcjOzhYACKdOnTJ5XyJqWhwzQC3Wjh074ObmhoqKCuj1egwfPhwJCQmG4127djUaJ3D69GlkZWXB3d3d6DplZWW4ePEiiouLkZ+fj7CwMMMxBwcHPPTQQzW6CqplZGTA3t4ejz/+uNlxZ2Vl4datW3jiiSeM9mu1Wjz44IMAgPPnzxvFAQDh4eFm36Papk2bsGLFCly8eBElJSWorKyEXC43atO2bVu0adPG6D56vR6ZmZlwd3fHxYsXMXbsWIwfP97QprKyEgqFwuJ4iKhpMBmgFisyMhKrV6+GVCqFSqWCg4Pxr7urq6vR65KSEvTq1QvJyck1rtW6det6xeDs7GzxOSUlJQCAnTt3Gn0IA1XjIGwlLS0NI0aMwPz58xEdHQ2FQoGNGzdiyZIlFsf66aef1khO7O3tbRYrETUsJgPUYrm6uiI4ONjs9j179sSmTZvg4+NT49txNT8/Pxw7dgx9+vQBUPUNOD09HT179qy1fdeuXaHX63Hw4EH069evxvHqyoROpzPsCw0NhUwmQ05OjsmKQufOnQ2DIasdPXq07jd5lyNHjiAwMBBvvPGGYd+VK1dqtMvJyUFeXh5UKpXhPnZ2dggJCYGvry9UKhUuXbqEESNGWHR/Imo+OICQ6E8jRoyAt7c3Bg8ejEOHDiE7OxsHDhzA5MmTcfXqVQDAq6++infffRdbt27FhQsX8Morr9xzjYB27dohJiYGY8aMwdatWw3X3Lx5MwAgMDAQEokEO3bswPXr11FSUgJ3d3dMmzYN8fHx+Pzzz3Hx4kWcPHkSH374oWFQ3sSJE/Hrr79i+vTpyMzMxIYNG5CUlGTR+/3LX/6CnJwcbNy4ERcvXsSKFStqHQzp5OSEmJgYnD59GocOHcLkyZPx7LPPQqlUAgDmz5+PxMRErFixAr/88gt++uknrF+/HkuXLrUoHiJqOkwGiP7k4uKC1NRUtG3bFkOHDkXnzp0xduxYlJWVGSoFr732Gl544QXExMQgPDwc7u7uePrpp+953dWrV+OZZ57BK6+8gk6dOmH8+PEoLS0FALRp0wbz58/HrFmz4Ovri7i4OADAwoULMWfOHCQmJqJz587o378/du7ciaCgIABV/fhff/01tm7diu7du2PNmjV45513LHq/Tz31FOLj4xEXF4cePXrgyJEjmDNnTo12wcHBGDp0KJ588klERUWhW7duRlMHx40bh7Vr12L9+vXo2rUrHn/8cSQlJRliJaLmTyKYGvlEREREosDKABERkcgxGSAiIhI5JgNEREQix2SAiIhI5JgMEBERiRyTASIiIpFjMkBERCRyTAaIiIhEjskAERGRyDEZICIiEjkmA0RERCL3/30OWm9j02xqAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "y_clean = df_clean['Experimental value'].astype(int)\n", "y_vega_clean = df_clean['Predicted value'].astype(int)\n", "\n", "cm = confusion_matrix(y_clean, y_vega_clean)\n", "disp = ConfusionMatrixDisplay(confusion_matrix=cm)\n", "disp.plot()\n", "print('{:<10} {:<15}'.format('Accuracy:', accuracy_score(y_clean, y_vega_clean)))\n", "print('{:<10} {:<15}'.format('Precision:', precision_score(y_clean, y_vega_clean)))\n", "print('{:<10} {:<15}'.format('Recall:', recall_score(y_clean, y_vega_clean)))\n", "print('{:<10} {:<15}'.format('F1:', f1_score(y_clean, y_vega_clean)))" ] }, { "cell_type": "markdown", "metadata": { "id": "x-imyMSBUGaE" }, "source": [ "#### Exercise - leave-one-out cross-validation ❗❗\n", "\n", "* Can you now assess our previous kNN model configuration using the leave-one-out approach on the cleaned dataset?" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "uc3jhsfoUZOD", "outputId": "e2208f4e-2555-420e-f533-944d6f46e230" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of folds: 5758\n" ] } ], "source": [ "from sklearn.model_selection import LeaveOneOut\n", "\n", "X_clean = df_clean.drop(['Unnamed: 0', 'Id','CAS','SMILES','Status','Experimental value','Predicted value'],axis=1)\n", "\n", "loo = LeaveOneOut()\n", "print('Number of folds: ', loo.get_n_splits(X_clean))" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "id": "irPFyGSuW5H-" }, "outputs": [], "source": [ "y_pred_loo_our_model = np.zeros(y_clean.shape[0])\n", "for i, (train_index, test_index) in enumerate(loo.split(X_clean)):\n", " # Get training data for the current fold\n", " X_train_loo = X_clean.iloc[train_index]\n", " y_train_loo = y_clean.iloc[train_index]\n", "\n", " # Get test data for the current fold\n", " X_test_loo = X_clean.iloc[test_index]\n", "\n", " # Train kNN\n", " # Your code here\n", "\n", " # Get prediction on the test molecule\n", " # Your code here\n", "\n", " # Store the prediction in `y_pred_loo_our_model`\n", " # Your code here\n" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 403 }, "id": "6FW3Wk0KYWnC", "outputId": "ee03b470-b467-4240-a3b9-040330470fcb" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy: 0.4357415769364363\n", "Precision: 0.0 \n", "Recall: 0.0 \n", "F1: 0.0 \n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "C:\\Users\\DELL\\Anaconda3\\envs\\ML4ChemEng\\lib\\site-packages\\sklearn\\metrics\\_classification.py:1318: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.\n", " _warn_prf(average, modifier, msg_start, len(result))\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAGwCAYAAAA0bWYRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA9p0lEQVR4nO3de3gU9dn/8c9mk80BsoEASYiECFKBVA6KFtIqQo0EpCgFn1ZFQQSsNlgBBeRXOVfTBw8oilBFjfSBCp6ogIIRykmilmAUEVKBWEBIACEJieS0O78/YlZXWMlmEzbZeb+uay7dme/M3ssV2Dv3/Z3vWAzDMAQAAEwryN8BAAAA/yIZAADA5EgGAAAwOZIBAABMjmQAAACTIxkAAMDkSAYAADC5YH8H4Aun06kjR44oMjJSFovF3+EAALxkGIZOnz6t+Ph4BQU13O+nZWVlqqio8Pk6NptNYWFh9RBR49Kkk4EjR44oISHB32EAAHx06NAhtWvXrkGuXVZWpg6JzZV/zOHzteLi4pSXlxdwCUGTTgYiIyMlSXes/a1szUL8HA3QMPb3K/d3CECDqVKltukd17/nDaGiokL5xxz6b/bFskfWvfpQfNqpxF5fqaKigmSgMalpDdiahcjWnGQAgSnY4vR3CEDD+W5B/AvR6m0eaVHzyLq/j1OB245u0skAAAC15TCccvjwNB6HEbiJOckAAMAUnDLkVN2zAV/Obey4tRAAAJOjMgAAMAWnnPKl0O/b2Y0byQAAwBQchiGHUfdSvy/nNna0CQAAMDkqAwAAU2ACoWckAwAAU3DKkINk4JxoEwAAYHJUBgAApkCbwDOSAQCAKXA3gWe0CQAAMDkqAwAAU3B+t/lyfqAiGQAAmILDx7sJfDm3sSMZAACYgsOQj08trL9YGhvmDAAAYHJUBgAApsCcAc9IBgAApuCURQ5ZfDo/UNEmAADA5KgMAABMwWlUb76cH6hIBgAApuDwsU3gy7mNHW0CAABMjmQAAGAKNZUBXzZvLFq0SN27d5fdbpfdbldycrLeffdd1/GysjKlpaWpVatWat68uYYPH66CggK3axw8eFCDBw9WRESEYmJiNHnyZFVVVbmN2bRpk6644gqFhoaqU6dOysjI8PrPhmQAAGAKTsPi8+aNdu3a6a9//auys7O1Y8cO/frXv9ZNN92k3bt3S5ImTpyo1atX67XXXtPmzZt15MgRDRs2zHW+w+HQ4MGDVVFRoe3bt+uVV15RRkaGZsyY4RqTl5enwYMHq3///srJydGECRM0duxYrV+/3qtYLYbRdB/DVFxcrKioKI3Z9DvZmof4OxygQXx5Vbm/QwAaTJVRqU36p4qKimS32xvkPWq+K7Z9Hq/mkXX/HbjktFNXX3ZEhw4dcos1NDRUoaGhtbpGdHS0HnvsMd18881q06aNli9frptvvlmStHfvXnXt2lVZWVnq06eP3n33Xf3mN7/RkSNHFBsbK0lavHixpk6dquPHj8tms2nq1Klau3atPv/8c9d73HLLLSosLNS6detq/dmoDAAATKG+2gQJCQmKiopybenp6ed/b4dDr776qkpLS5WcnKzs7GxVVlYqJSXFNaZLly5q3769srKyJElZWVnq1q2bKxGQpNTUVBUXF7uqC1lZWW7XqBlTc43a4m4CAIApOBQkhw+/Azu++++5KgOe7Nq1S8nJySorK1Pz5s311ltvKSkpSTk5ObLZbGrRooXb+NjYWOXn50uS8vPz3RKBmuM1x35qTHFxsc6cOaPw8PBafTaSAQCAKRh16Pv/+HxJrgmBtdG5c2fl5OSoqKhIr7/+ukaNGqXNmzfXOYaGQjIAAEADsdls6tSpkySpV69e+ve//62nn35av//971VRUaHCwkK36kBBQYHi4uIkSXFxcfr444/drldzt8EPx/z4DoSCggLZ7fZaVwUk5gwAAEziQt9aeC5Op1Pl5eXq1auXQkJCtGHDBtex3NxcHTx4UMnJyZKk5ORk7dq1S8eOHXONyczMlN1uV1JSkmvMD69RM6bmGrVFZQAAYAoOI0gOw4c5A17eezdt2jQNGjRI7du31+nTp7V8+XJt2rRJ69evr74TbswYTZo0SdHR0bLb7brvvvuUnJysPn36SJIGDBigpKQk3XHHHZo3b57y8/P18MMPKy0tzTVP4Z577tGzzz6rKVOm6K677tLGjRu1cuVKrV271qtYSQYAAGgAx44d08iRI3X06FFFRUWpe/fuWr9+va6//npJ0vz58xUUFKThw4ervLxcqampeu6551znW61WrVmzRvfee6+Sk5PVrFkzjRo1SnPmzHGN6dChg9auXauJEyfq6aefVrt27bRkyRKlpqZ6FSvrDACNHOsMIJBdyHUG1n7WUc0irXW+TulphwZ3P9CgsfoLlQEAgCnwoCLPmEAIAIDJURkAAJiC7xMIm2xX/bxIBgAApuCURU4fSv2+nNvY0SYAAMDkqAwAAEzB6eOzCZyiTQAAQJPGnAHPSAYAAKbgVJCcVAbOiTkDAACYHJUBAIApOAyLHD48wtiXcxs7kgEAgCk4fJxA6KBNAAAAAhWVAQCAKTiNIDl9uJvAyd0EAAA0bbQJPKNNAACAyVEZAACYglO+3RHgrL9QGh2SAQCAKfi+6FDgFtMD95MBAIBaoTIAADAF359NELi/P5MMAABMwSmLnPJlzgArEAIA0KRRGfAscD8ZAACoFSoDAABT8H3RocD9/ZlkAABgCk7DIqcv6wwE8FMLAzfNAQAAtUJlAABgCk4f2wSBvOgQyQAAwBR8f2ph4CYDgfvJAABArVAZAACYgkMWOXxYOMiXcxs7kgEAgCnQJvAscD8ZAACoFSoDAABTcMi3Ur+j/kJpdEgGAACmQJvAM5IBAIAp8KAizwL3kwEAgFqhMgAAMAVDFjl9mDNgcGshAABNG20CzwL3kwEAgFqhMgAAMAUeYewZyQAAwBQcPj610JdzG7vA/WQAAKBWqAwAAEyBNoFnJAMAAFNwKkhOHwrivpzb2AXuJwMAALVCZQAAYAoOwyKHD6V+X85t7EgGAACmwJwBz2gTAABMwfjuqYV13QwvVyBMT0/XVVddpcjISMXExGjo0KHKzc11G9OvXz9ZLBa37Z577nEbc/DgQQ0ePFgRERGKiYnR5MmTVVVV5TZm06ZNuuKKKxQaGqpOnTopIyPDq1hJBgAAaACbN29WWlqaPvzwQ2VmZqqyslIDBgxQaWmp27hx48bp6NGjrm3evHmuYw6HQ4MHD1ZFRYW2b9+uV155RRkZGZoxY4ZrTF5engYPHqz+/fsrJydHEyZM0NixY7V+/fpax0qbAABgCg5Z5PDhYUM15xYXF7vtDw0NVWho6Fnj161b5/Y6IyNDMTExys7OVt++fV37IyIiFBcXd873fO+99/TFF1/o/fffV2xsrHr27Km5c+dq6tSpmjVrlmw2mxYvXqwOHTroiSeekCR17dpV27Zt0/z585Wamlqrz0ZlAABgCk7j+3kDdduqr5OQkKCoqCjXlp6eXqv3LyoqkiRFR0e77V+2bJlat26tyy67TNOmTdO3337rOpaVlaVu3bopNjbWtS81NVXFxcXavXu3a0xKSorbNVNTU5WVlVXrPxsqAwAAeOHQoUOy2+2u1+eqCvyY0+nUhAkT9Ktf/UqXXXaZa/9tt92mxMRExcfH67PPPtPUqVOVm5urN998U5KUn5/vlghIcr3Oz8//yTHFxcU6c+aMwsPDzxsfyYDJnHy5SiX/cqriv4aCQqWw7kFqPd4q28XfF4kO/6FCZ3YabufZhwUpdlqI63VlvqFjf63SmR1OBUVIkYOtap1mlSX4+xJc4UqHCl9zqOqooeBYi6Lvsso+2NrwHxKooyF3ntDN9x5TdJsqHfgiXM89fJFycyL8HRbqSc1EQF/OlyS73e6WDNRGWlqaPv/8c23bts1t/9133+36/27duqlt27a67rrrtH//fl1yySV1jtVbJAMmc2anUy3+x6rQJIvkkE4859DX91UqcaVNQeHff5Hbhwap1R++//GwhH1/DcNh6MiESllbSQkvhqjqhKGCWVWyBEut06rPKXzdoW+eq1LM/wtWWFKQyr5wquCRKgVFSs37khCg8bn2xlO6e+YRPfNQO+3dGaHfjjuuR5Yf0JhrOqvom5DzXwCNnlMWOX2YM1DXc8ePH681a9Zoy5Ytateu3U+O7d27tyRp3759uuSSSxQXF6ePP/7YbUxBQYEkueYZxMXFufb9cIzdbq9VVUBqJHMGFi5cqIsvvlhhYWHq3bv3WR8c9eeiZ2yyD7Eq9JIghV4apNiZwarKl8r3uFcCgsIsCm79/WZt/v1fgm8/dKoiz1DcnBCFdg5Ss19ZFX1PsIpec8iorL7O6Xccsv/WqsgBVoW0syhygFVRQ606tdRxQT8vUFvD7j6hdcuj9d6KaB38MkwLprZT+RmLUm896e/Q0EQZhqHx48frrbfe0saNG9WhQ4fznpOTkyNJatu2rSQpOTlZu3bt0rFjx1xjMjMzZbfblZSU5BqzYcMGt+tkZmYqOTm51rH6PRlYsWKFJk2apJkzZ2rnzp3q0aOHUlNT3T44Go6zpPq/QT+qeJ1e59D+lHL99/cVOvFslZxl3ycLZbsM2S6xKLjV9wlCsz5BcpZK5QeqxxmVUpDN/ZqWMKlstyGjyj3xAPwtOMSpn3X/Vju3Rrr2GYZFn2yNVFKvb3/iTDQlNSsQ+rJ5Iy0tTf/3f/+n5cuXKzIyUvn5+crPz9eZM2ckSfv379fcuXOVnZ2tr776Sm+//bZGjhypvn37qnv37pKkAQMGKCkpSXfccYc+/fRTrV+/Xg8//LDS0tJccxXuueceHThwQFOmTNHevXv13HPPaeXKlZo4cWKtY/V7MvDkk09q3LhxGj16tJKSkrR48WJFRETopZde8ndoAc9wGjr+ZJXCelgU2un7H4XIVKti54So3eIQtbzTqtPvOpQ//fsFLqq+MRTcyv1a1u9eO05Uf9FH9AlS0T8dKtvjlGEYKvvCqeJVDqlKchQ29CcDvGOPdsgaLBUed++cnjoRrJZtqjychabGlwWH6jLfYNGiRSoqKlK/fv3Utm1b17ZixQpJks1m0/vvv68BAwaoS5cueuCBBzR8+HCtXr3adQ2r1ao1a9bIarUqOTlZt99+u0aOHKk5c+a4xnTo0EFr165VZmamevTooSeeeEJLliyp9W2Fkp/nDFRUVCg7O1vTpk1z7QsKClJKSso5b4koLy9XeXm56/WP7/WEd47Pq1LFfqfaveD+K3zUsO97+qGdpODWFn39x0pVHDZka1e7zDh6jFWObwwdGl0pSbJGS/bffNcmCNwVPQHAxTB+ugqakJCgzZs3n/c6iYmJeuedd35yTL9+/fTJJ594Fd8P+bUycOLECTkcjnPeElFzy8QPpaenu93bmZCQcKFCDTjH5lWqdKtT7RbZFBL709/OYZdVH688VP2DHdzKoqpv3Mc4vnttbV09NijMotgZIeq0zaaL/2lTh9U2Bbe1KKiZZG1Zv58F8FXxSascVVKLH1UBWrau0qnjzLMOFE75ssaAb5MPGzu/twm8MW3aNBUVFbm2Q4cO+TukJscwDB2bV6mSTU5dtChEIRed/4e7/D/fJQGtq1+HdbOoYr+hqpPfZ73ffuRUUDPJ1sH9epZgi0JiLbJYLSp5z6GIq4NkCQrcv1Bomqoqg/TlZxG6/OrTrn0Wi6GeV5foi2xuLQwUxnd3E9R1MwI4GfBrytu6dWtZrdZz3hJxrqUZPS35iNo7/r9VOr3eqbaPhygowqKq73r8Qc2rf5uvOGzo9DqHmv0qSNYoi8q/dOrE/CqFX25R6M+qc8eIPkGydbCoYGalWt8XrKpvpG8WVynqf6wKslX/Zan4r1Nluw2FXWaR87R0aplD5QcMtZ/FLVponN58vrUefOqQ/vNphHI/qb61MCzCqfdejT7/yWgSeGqhZ35NBmw2m3r16qUNGzZo6NChkqpXadqwYYPGjx/vz9ACVtEbTknS1/dUuu2PnREs+xCrLMHSmY+dKnzVIeOMFBxrUfNfW9Xyru/nEVisFsXPD9Gxv1bp0F2VCgqvXnSo1R9+sH6AUypc5lDFfw1ZgqXwK4OUsCREIfGB+5cJTdvmt1sqqpVDIyfnq2WbKh3YHa4/j+igwhMksAh8fm+GTZo0SaNGjdKVV16pX/ziF3rqqadUWlqq0aNH+zu0gPSzf/90ZSUkzqJ2z9t+cowkhbS16KKnPf8jaesQpPbLzn8doDF5++XWevvl1v4OAw2kvlYgDER+TwZ+//vf6/jx45oxY4by8/PVs2dPrVu37qxJhQAA+II2gWd+Twak6qUaaQsAAOAfjSIZAACgofnr2QRNAckAAMAUaBN4FrizIQAAQK1QGQAAmAKVAc9IBgAApkAy4BltAgAATI7KAADAFKgMeEYyAAAwBUO+3R740w8kbtpIBgAApkBlwDPmDAAAYHJUBgAApkBlwDOSAQCAKZAMeEabAAAAk6MyAAAwBSoDnpEMAABMwTAsMnz4Qvfl3MaONgEAACZHZQAAYApOWXxadMiXcxs7kgEAgCkwZ8Az2gQAAJgclQEAgCkwgdAzkgEAgCnQJvCMZAAAYApUBjxjzgAAACZHZQAAYAqGj22CQK4MkAwAAEzBkGQYvp0fqGgTAABgclQGAACm4JRFFlYgPCeSAQCAKXA3gWe0CQAAMDkqAwAAU3AaFllYdOicSAYAAKZgGD7eTRDAtxPQJgAAwOSoDAAATIEJhJ6RDAAATIFkwDOSAQCAKTCB0DPmDAAAYHJUBgAApsDdBJ6RDAAATKE6GfBlzkA9BtPI0CYAAMDkSAYAAKZQczeBL5s30tPTddVVVykyMlIxMTEaOnSocnNz3caUlZUpLS1NrVq1UvPmzTV8+HAVFBS4jTl48KAGDx6siIgIxcTEaPLkyaqqqnIbs2nTJl1xxRUKDQ1Vp06dlJGR4VWsJAMAAFMw6mHzxubNm5WWlqYPP/xQmZmZqqys1IABA1RaWuoaM3HiRK1evVqvvfaaNm/erCNHjmjYsGGu4w6HQ4MHD1ZFRYW2b9+uV155RRkZGZoxY4ZrTF5engYPHqz+/fsrJydHEyZM0NixY7V+/fpax2oxjKbbBSkuLlZUVJTGbPqdbM1D/B0O0CC+vKrc3yEADabKqNQm/VNFRUWy2+0N8h413xWX/H2arBFhdb6O49sy7b8jXYcOHXKLNTQ0VKGhoec9//jx44qJidHmzZvVt29fFRUVqU2bNlq+fLluvvlmSdLevXvVtWtXZWVlqU+fPnr33Xf1m9/8RkeOHFFsbKwkafHixZo6daqOHz8um82mqVOnau3atfr8889d73XLLbeosLBQ69atq9VnozIAADCF+moTJCQkKCoqyrWlp6fX6v2LiookSdHR0ZKk7OxsVVZWKiUlxTWmS5cuat++vbKysiRJWVlZ6tatmysRkKTU1FQVFxdr9+7drjE/vEbNmJpr1AZ3EwAAzKEutf4fny+dszJwPk6nUxMmTNCvfvUrXXbZZZKk/Px82Ww2tWjRwm1sbGys8vPzXWN+mAjUHK859lNjiouLdebMGYWHh583PpIBAIA5+Lgcsb471263e93SSEtL0+eff65t27bV/f0bEG0CAAAa0Pjx47VmzRr961//Urt27Vz74+LiVFFRocLCQrfxBQUFiouLc4358d0FNa/PN8Zut9eqKiCRDAAATKJmBUJfNu/ez9D48eP11ltvaePGjerQoYPb8V69eikkJEQbNmxw7cvNzdXBgweVnJwsSUpOTtauXbt07Ngx15jMzEzZ7XYlJSW5xvzwGjVjaq5RG7QJAACmcKGfWpiWlqbly5frn//8pyIjI109/qioKIWHh1ffDTdmjCZNmqTo6GjZ7Xbdd999Sk5OVp8+fSRJAwYMUFJSku644w7NmzdP+fn5evjhh5WWluaaq3DPPffo2Wef1ZQpU3TXXXdp48aNWrlypdauXVvrWKkMAADQABYtWqSioiL169dPbdu2dW0rVqxwjZk/f75+85vfaPjw4erbt6/i4uL05ptvuo5brVatWbNGVqtVycnJuv322zVy5EjNmTPHNaZDhw5au3atMjMz1aNHDz3xxBNasmSJUlNTax0rlQEAgDkYFtckwDqf783wWvQVwsLCtHDhQi1cuNDjmMTERL3zzjs/eZ1+/frpk08+8Sq+HyIZAACYAk8t9Iw2AQAAJkdlAABgDvW06FAgIhkAAJjChb6boCmpVTLw9ttv1/qCN954Y52DAQAAF16tkoGhQ4fW6mIWi0UOh8OXeAAAaDgBXOr3Ra2SAafT2dBxAADQoGgTeObT3QRlZWX1FQcAAA3LqIctQHmdDDgcDs2dO1cXXXSRmjdvrgMHDkiSpk+frhdffLHeAwQAAA3L62TgkUceUUZGhubNmyebzebaf9lll2nJkiX1GhwAAPXHUg9bYPI6GVi6dKmef/55jRgxQlar1bW/R48e2rt3b70GBwBAvaFN4JHXycDXX3+tTp06nbXf6XSqsrKyXoICAAAXjtfJQFJSkrZu3XrW/tdff12XX355vQQFAEC9ozLgkdcrEM6YMUOjRo3S119/LafTqTfffFO5ublaunSp1qxZ0xAxAgDguwv81MKmxOvKwE033aTVq1fr/fffV7NmzTRjxgzt2bNHq1ev1vXXX98QMQIAgAZUp2cTXHPNNcrMzKzvWAAAaDA8wtizOj+oaMeOHdqzZ4+k6nkEvXr1qregAACodzy10COvk4HDhw/r1ltv1QcffKAWLVpIkgoLC/XLX/5Sr776qtq1a1ffMQIAgAbk9ZyBsWPHqrKyUnv27NHJkyd18uRJ7dmzR06nU2PHjm2IGAEA8F3NBEJftgDldWVg8+bN2r59uzp37uza17lzZz3zzDO65ppr6jU4AADqi8Wo3nw5P1B5nQwkJCScc3Ehh8Oh+Pj4egkKAIB6x5wBj7xuEzz22GO67777tGPHDte+HTt26P7779fjjz9er8EBAICGV6vKQMuWLWWxfN8rKS0tVe/evRUcXH16VVWVgoODddddd2no0KENEigAAD5h0SGPapUMPPXUUw0cBgAADYw2gUe1SgZGjRrV0HEAAAA/qfOiQ5JUVlamiooKt312u92ngAAAaBBUBjzyegJhaWmpxo8fr5iYGDVr1kwtW7Z02wAAaJR4aqFHXicDU6ZM0caNG7Vo0SKFhoZqyZIlmj17tuLj47V06dKGiBEAADQgr9sEq1ev1tKlS9WvXz+NHj1a11xzjTp16qTExEQtW7ZMI0aMaIg4AQDwDXcTeOR1ZeDkyZPq2LGjpOr5ASdPnpQkXX311dqyZUv9RgcAQD2pWYHQly1QeZ0MdOzYUXl5eZKkLl26aOXKlZKqKwY1Dy4CAABNh9fJwOjRo/Xpp59Kkh566CEtXLhQYWFhmjhxoiZPnlzvAQIAUC+YQOiR13MGJk6c6Pr/lJQU7d27V9nZ2erUqZO6d+9er8EBAICG59M6A5KUmJioxMTE+ogFAIAGY5GPTy2st0gan1olAwsWLKj1Bf/0pz/VORgAAHDh1SoZmD9/fq0uZrFY/JIMPB6/Q/ZIr6c/AE1Cqnr6OwQgMHBroUe1SgZq7h4AAKDJYjlij/h1GgAAk/N5AiEAAE0ClQGPSAYAAKbg6yqCrEAIAAACFpUBAIA50CbwqE6Vga1bt+r2229XcnKyvv76a0nS3//+d23btq1egwMAoN6wHLFHXicDb7zxhlJTUxUeHq5PPvlE5eXlkqSioiI9+uij9R4gAABoWF4nA3/5y1+0ePFivfDCCwoJCXHt/9WvfqWdO3fWa3AAANQXHmHsmdfJQG5urvr27XvW/qioKBUWFtZHTAAA1L+aFQh92bywZcsWDRkyRPHx8bJYLFq1apXb8TvvvFMWi8VtGzhwoNuYkydPasSIEbLb7WrRooXGjBmjkpIStzGfffaZrrnmGoWFhSkhIUHz5s3z+o/G62QgLi5O+/btO2v/tm3b1LFjR68DAADggrjAcwZKS0vVo0cPLVy40OOYgQMH6ujRo67tH//4h9vxESNGaPfu3crMzNSaNWu0ZcsW3X333a7jxcXFGjBggBITE5Wdna3HHntMs2bN0vPPP+9VrF7fTTBu3Djdf//9eumll2SxWHTkyBFlZWXpwQcf1PTp0729HAAAAWnQoEEaNGjQT44JDQ1VXFzcOY/t2bNH69at07///W9deeWVkqRnnnlGN9xwgx5//HHFx8dr2bJlqqio0EsvvSSbzaaf//znysnJ0ZNPPumWNJyP15WBhx56SLfddpuuu+46lZSUqG/fvho7dqz+8Ic/6L777vP2cgAAXBD1NWeguLjYbauZSF8XmzZtUkxMjDp37qx7771X33zzjetYVlaWWrRo4UoEJCklJUVBQUH66KOPXGP69u0rm83mGpOamqrc3FydOnWq1nF4nQxYLBb9+c9/1smTJ/X555/rww8/1PHjxzV37lxvLwUAwIVTT22ChIQERUVFubb09PQ6hTNw4EAtXbpUGzZs0P/+7/9q8+bNGjRokBwOhyQpPz9fMTExbucEBwcrOjpa+fn5rjGxsbFuY2pe14ypjTovOmSz2ZSUlFTX0wEAaJIOHToku93ueh0aGlqn69xyyy2u/+/WrZu6d++uSy65RJs2bdJ1113nc5ze8DoZ6N+/vywWzzMqN27c6FNAAAA0CF9vD/zuXLvd7pYM1JeOHTuqdevW2rdvn6677jrFxcXp2LFjbmOqqqp08uRJ1zyDuLg4FRQUuI2pee1pLsK5eN0m6Nmzp3r06OHakpKSVFFRoZ07d6pbt27eXg4AgAujka9AePjwYX3zzTdq27atJCk5OVmFhYXKzs52jdm4caOcTqd69+7tGrNlyxZVVla6xmRmZqpz585q2bJlrd/b68rA/Pnzz7l/1qxZZ937CACAWZWUlLjdip+Xl6ecnBxFR0crOjpas2fP1vDhwxUXF6f9+/drypQp6tSpk1JTUyVJXbt21cCBAzVu3DgtXrxYlZWVGj9+vG655RbFx8dLkm677TbNnj1bY8aM0dSpU/X555/r6aef9vhd7Um9PbXw9ttv10svvVRflwMAoH5d4MrAjh07dPnll+vyyy+XJE2aNEmXX365ZsyYIavVqs8++0w33nijLr30Uo0ZM0a9evXS1q1b3eYgLFu2TF26dNF1112nG264QVdffbXbGgJRUVF67733lJeXp169eumBBx7QjBkzvLqtUKrHpxZmZWUpLCysvi4HAEC98nVJYW/P7devnwzD80nr168/7zWio6O1fPnynxzTvXt3bd261bvgfsTrZGDYsGFurw3D0NGjR7Vjxw4WHQIAoAnyOhmIiopyex0UFKTOnTtrzpw5GjBgQL0FBgAALgyvkgGHw6HRo0erW7duXs1SBADA73y9I4CnFlazWq0aMGAATycEADQ5PMLYM6/vJrjssst04MCBhogFAAD4gdfJwF/+8hc9+OCDWrNmjY4ePXrWAxsAAGi0GumCQ/5W6zkDc+bM0QMPPKAbbrhBknTjjTe6LUtsGIYsFovrAQsAADQqzBnwqNbJwOzZs3XPPffoX//6V0PGAwAALrBaJwM1Cydce+21DRYMAAAN5UIvOtSUeHVr4U89rRAAgEaNNoFHXiUDl1566XkTgpMnT/oUEAAAuLC8SgZmz5591gqEAAA0BbQJPPMqGbjlllsUExPTULEAANBwaBN4VOt1BpgvAABAYPL6bgIAAJokKgMe1ToZcDqdDRkHAAANijkDnnn9CGMAAJokKgMeef1sAgAAEFioDAAAzIHKgEckAwAAU2DOgGe0CQAAMDkqAwAAc6BN4BHJAADAFGgTeEabAAAAk6MyAAAwB9oEHpEMAADMgWTAI9oEAACYHJUBAIApWL7bfDk/UJEMAADMgTaBRyQDAABT4NZCz5gzAACAyVEZAACYA20Cj0gGAADmEcBf6L6gTQAAgMlRGQAAmAITCD0jGQAAmANzBjyiTQAAgMlRGQAAmAJtAs9IBgAA5kCbwCPaBAAAmByVAQCAKdAm8IxkAABgDrQJPCIZAACYA8mAR8wZAADA5KgMAABMgTkDnpEMAADMgTaBR7QJAABoAFu2bNGQIUMUHx8vi8WiVatWuR03DEMzZsxQ27ZtFR4erpSUFH355ZduY06ePKkRI0bIbrerRYsWGjNmjEpKStzGfPbZZ7rmmmsUFhamhIQEzZs3z+tYSQYAAKZgMQyfN2+UlpaqR48eWrhw4TmPz5s3TwsWLNDixYv10UcfqVmzZkpNTVVZWZlrzIgRI7R7925lZmZqzZo12rJli+6++27X8eLiYg0YMECJiYnKzs7WY489plmzZun555/3KlbaBAAAc7jAbYJBgwZp0KBB576UYeipp57Sww8/rJtuukmStHTpUsXGxmrVqlW65ZZbtGfPHq1bt07//ve/deWVV0qSnnnmGd1www16/PHHFR8fr2XLlqmiokIvvfSSbDabfv7znysnJ0dPPvmkW9JwPlQGAADwQnFxsdtWXl7u9TXy8vKUn5+vlJQU176oqCj17t1bWVlZkqSsrCy1aNHClQhIUkpKioKCgvTRRx+5xvTt21c2m801JjU1Vbm5uTp16lSt4yEZAACYQs3dBL5skpSQkKCoqCjXlp6e7nUs+fn5kqTY2Fi3/bGxsa5j+fn5iomJcTseHBys6OhotzHnusYP36M2aBMAAMyhntoEhw4dkt1ud+0ODQ31KazGgMoAAABesNvtbltdkoG4uDhJUkFBgdv+goIC17G4uDgdO3bM7XhVVZVOnjzpNuZc1/jhe9QGyQAAwBTqq01QHzp06KC4uDht2LDBta+4uFgfffSRkpOTJUnJyckqLCxUdna2a8zGjRvldDrVu3dv15gtW7aosrLSNSYzM1OdO3dWy5Ytax0PyQAAwByMeti8UFJSopycHOXk5EiqnjSYk5OjgwcPymKxaMKECfrLX/6it99+W7t27dLIkSMVHx+voUOHSpK6du2qgQMHaty4cfr444/1wQcfaPz48brlllsUHx8vSbrttttks9k0ZswY7d69WytWrNDTTz+tSZMmeRUrcwYAAKZwoZcj3rFjh/r37+96XfMFPWrUKGVkZGjKlCkqLS3V3XffrcLCQl199dVat26dwsLCXOcsW7ZM48eP13XXXaegoCANHz5cCxYscB2PiorSe++9p7S0NPXq1UutW7fWjBkzvLqtsPqzGV6uotCIFBcXKyoqSqf+01H2SIocCEyp8T39HQLQYKqMSm3SP1VUVOQ2Ka8+1XxX9Pr9I7Laws5/ggeOijJlr/hzg8bqL1QGAADmwLMJPCIZAACYRiA/edAX1NYBADA5KgMAAHMwjOrNl/MDFMkAAMAULvTdBE0JbQIAAEyOygAAwBy4m8AjkgEAgClYnNWbL+cHKtoEAACYHJUBk1n9SiutXdpaBYdskqTEzmUaMTFfV/36tIpPWfX3x+O0c3Okjh2xKSq6Sr8cWKRRU46qmf3slLj4pFX3Xt9ZJ47a9MaeXWoe5XAde/vl1nr75dYqOGxTTHyFbrm/QNf/z6kL9jmBuhhy5wndfO8xRbep0oEvwvXcwxcpNyfC32GhvtAm8IhkwGTatK3UXf/viC7qUC7DsCjztZaaNbqDFr73H8mQvikI0bgZR9T+0jIdO2zTgofa6ZuCEE1/4auzrvXkA+3VoWuZThy1ue1f/UorvZzeVvc/dkide36r3E8i9NTkBEVGOdRnQPEF+qSAd6698ZTunnlEzzzUTnt3Rui3447rkeUHNOaazir6JsTf4aEecDeBZ35tE2zZskVDhgxRfHy8LBaLVq1a5c9wTKHPgGL94rrTuqhjhdpdUq7RD+UrrJlTe7MjdHGXMs1Y8pX6DChW/MUV6nl1ie6celQfZdrlqHK/zupXWqm02Kqb7zl21ntseD1aN9z+jfrdVKi2iRXqN7RQg27/RisXxlygTwl4b9jdJ7RuebTeWxGtg1+GacHUdio/Y1HqrSf9HRrqS806A75sAcqvyUBpaal69OihhQsX+jMM03I4pE2rWqj82yB1vbL0nGNKi62KaO6U9Qc1pP/+J1TL58dp8tP/leUcP0GVFRbZwtzbCqFhTuXmRKiq8uzxgL8Fhzj1s+7faufWSNc+w7Dok62RSur1rR8jAy4Mv7YJBg0apEGDBtV6fHl5ucrLy12vi4spOddF3p4wTRjyM1WUBym8mVMzXsxT4qXlZ40r+saq5U/FadDtJ1z7KsotSv/jxRo7/Yhi2lXq6MHQs87r1e+01i1vpV8OLFKnbmf05WfhWre8laoqg1R0MlitYqvOOgfwJ3u0Q9ZgqfC4+z+Jp04EK6HT2X830DTRJvCsSc0ZSE9P1+zZs/0dRpPX7pJyPZeZq29PW7V1TQs9fn+iHnvzS7eEoPR0kKaP7Kj2l5bpjgfyXftfTm+r9p3KdN1wz5MBR0zI16ljwbr/N5fKMKSWbSqV8j8n9dpzsQri/hUA/sIEQo+aVDIwbdo0TZo0yfW6uLhYCQkJfoyoaQqxGbqoQ4Uk6Wfdzyg3J0KrlrTR/fMOS5K+LQnSn2+7ROHNnJr5Yp6CfzB3KmdbpL7aG6ZBCS2qd3z3l+N/LrtMt/6pQCMn5ys03NAD8w/p/nmHdOp4iKJjK/XO/7VSRHOHolpRFUDjU3zSKkeV1KKN+89ny9ZVOnW8Sf0zCdRJk/opDw0NVWjo2WVp+MYwpMqK6l/ZS09XJwIhNkOzMw7IFuaeCk9fkqeKsu9/vc/NidCTk9rribe+VPzFFW5jg0OkNvHVkwQ2/7OlfpFSTGUAjVJVZZC+/CxCl199WlnroiRJFouhnleX6O2MVn6ODvWFNoFnTSoZgO9eerStrvp1sdpcVKkzJUH611st9dn25npk+X6Vng7S/7v1EpWfCdKUZ/L0bYlV35ZUnxfVqkpWq876wi86Wf0j1P5n5a51Bg7vD1VuToS6XF6q00XBevNvbfRVbpgefPrgBf2sgDfefL61HnzqkP7zaYRyP6m+tTAswqn3Xo32d2ioLzy10COSAZMpPBGsx/6UqJPHghUR6VCHrmV6ZPl+9bq2RJ9ub669O5tJkkb/MsntvFc++kJxCRXnuuRZnE7pjcVtdHh/gqwhhnr8skTz//llrc8H/GHz2y0V1cqhkZPz1bJNlQ7sDtefR3RQ4QnWGEDg82syUFJSon379rle5+XlKScnR9HR0Wrfvr0fIwtck5485PFYj1+WaP2RHK+ud65z2v+sXM9l/qcO0QH+VbNyJgITbQLP/JoM7NixQ/3793e9rpkcOGrUKGVkZPgpKgBAQOJuAo/8mgz069dPRgD3YAAAaAqYMwAAMAXaBJ6RDAAAzMFpVG++nB+gSAYAAObAnAGPWAIGAACTozIAADAFi3ycM1BvkTQ+JAMAAHNgBUKPaBMAAGByVAYAAKbArYWekQwAAMyBuwk8ok0AAIDJURkAAJiCxTBk8WESoC/nNnYkAwAAc3B+t/lyfoCiTQAAgMlRGQAAmAJtAs9IBgAA5sDdBB6RDAAAzIEVCD1izgAAACZHZQAAYAqsQOgZyQAAwBxoE3hEmwAAAJOjMgAAMAWLs3rz5fxARTIAADAH2gQe0SYAAMDkqAwAAMyBRYc8ojIAADCFmuWIfdm8MWvWLFksFretS5curuNlZWVKS0tTq1at1Lx5cw0fPlwFBQVu1zh48KAGDx6siIgIxcTEaPLkyaqqqqqXP48fojIAAEAD+fnPf67333/f9To4+Puv3YkTJ2rt2rV67bXXFBUVpfHjx2vYsGH64IMPJEkOh0ODBw9WXFyctm/frqNHj2rkyJEKCQnRo48+Wq9xkgwAAMzBDxMIg4ODFRcXd9b+oqIivfjii1q+fLl+/etfS5Jefvllde3aVR9++KH69Omj9957T1988YXef/99xcbGqmfPnpo7d66mTp2qWbNmyWaz1f2z/AhtAgCAORiSnD5s3+UCxcXFblt5ebnHt/zyyy8VHx+vjh07asSIETp48KAkKTs7W5WVlUpJSXGN7dKli9q3b6+srCxJUlZWlrp166bY2FjXmNTUVBUXF2v37t318AfyPZIBAIAp1NecgYSEBEVFRbm29PT0c75f7969lZGRoXXr1mnRokXKy8vTNddco9OnTys/P182m00tWrRwOyc2Nlb5+fmSpPz8fLdEoOZ4zbH6RJsAAAAvHDp0SHa73fU6NDT0nOMGDRrk+v/u3burd+/eSkxM1MqVKxUeHt7gcXqDygAAwBwMfT9voE5b9WXsdrvb5ikZ+LEWLVro0ksv1b59+xQXF6eKigoVFha6jSkoKHDNMYiLizvr7oKa1+eah+ALkgEAgDn4lAj4OPlQUklJifbv36+2bduqV69eCgkJ0YYNG1zHc3NzdfDgQSUnJ0uSkpOTtWvXLh07dsw1JjMzU3a7XUlJST7F8mO0CQAAaAAPPvighgwZosTERB05ckQzZ86U1WrVrbfeqqioKI0ZM0aTJk1SdHS07Ha77rvvPiUnJ6tPnz6SpAEDBigpKUl33HGH5s2bp/z8fD388MNKS0urdTWitkgGAADm4JRk8fF8Lxw+fFi33nqrvvnmG7Vp00ZXX321PvzwQ7Vp00aSNH/+fAUFBWn48OEqLy9XamqqnnvuOdf5VqtVa9as0b333qvk5GQ1a9ZMo0aN0pw5c3z4EOdmMYym++SF4uJiRUVF6dR/OsoeSccDgSk1vqe/QwAaTJVRqU36p4qKitwm5dWnmu+K6y6bomBr3X+jrnKUa8Pn8xo0Vn/hGxQAAJOjTQAAMAceYewRyQAAwBxIBjyiTQAAgMlRGQAAmAOVAY9IBgAA5nCBby1sSkgGAACm8MOHDdX1/EDFnAEAAEyOygAAwByYM+ARyQAAwBychmTx4QvdGbjJAG0CAABMjsoAAMAcaBN4RDIAADAJH5MBBW4yQJsAAACTozIAADAH2gQekQwAAMzBacinUj93EwAAgEBFZQAAYA6Gs3rz5fwARTIAADAH5gx4RDIAADAH5gx4xJwBAABMjsoAAMAcaBN4RDIAADAHQz4mA/UWSaNDmwAAAJOjMgAAMAfaBB6RDAAAzMHplOTDWgHOwF1ngDYBAAAmR2UAAGAOtAk8IhkAAJgDyYBHtAkAADA5KgMAAHNgOWKPSAYAAKZgGE4ZPjx50JdzGzuSAQCAORiGb7/dM2cAAAAEKioDAABzMHycMxDAlQGSAQCAOTidksWHvn8AzxmgTQAAgMlRGQAAmANtAo9IBgAApmA4nTJ8aBME8q2FtAkAADA5KgMAAHOgTeARyQAAwBychmQhGTgX2gQAAJgclQEAgDkYhiRf1hkI3MoAyQAAwBQMpyHDhzaBQTIAAEATZzjlW2WAWwsBAEAdLFy4UBdffLHCwsLUu3dvffzxx/4O6SwkAwAAUzCchs+bt1asWKFJkyZp5syZ2rlzp3r06KHU1FQdO3asAT5h3ZEMAADMwXD6vnnpySef1Lhx4zR69GglJSVp8eLFioiI0EsvvdQAH7DumvScgZrJHMUlgdvHAaqMSn+HADSYKlX/fF+IyXlVqvRpzaGaWIuLi932h4aGKjQ09KzxFRUVys7O1rRp01z7goKClJKSoqysrLoH0gCadDJw+vRpSVLiFV/5NxCgQR3wdwBAgzt9+rSioqIa5No2m01xcXHalv+Oz9dq3ry5EhIS3PbNnDlTs2bNOmvsiRMn5HA4FBsb67Y/NjZWe/fu9TmW+tSkk4H4+HgdOnRIkZGRslgs/g7HFIqLi5WQkKBDhw7Jbrf7OxygXvHzfeEZhqHTp08rPj6+wd4jLCxMeXl5qqio8PlahmGc9X1zrqpAU9Okk4GgoCC1a9fO32GYkt1u5x9LBCx+vi+shqoI/FBYWJjCwsIa/H1+qHXr1rJarSooKHDbX1BQoLi4uAsay/kwgRAAgAZgs9nUq1cvbdiwwbXP6XRqw4YNSk5O9mNkZ2vSlQEAABqzSZMmadSoUbryyiv1i1/8Qk899ZRKS0s1evRof4fmhmQAXgkNDdXMmTMDokcG/Bg/36hvv//973X8+HHNmDFD+fn56tmzp9atW3fWpEJ/sxiBvNgyAAA4L+YMAABgciQDAACYHMkAAAAmRzIAAIDJkQyg1prCYziButiyZYuGDBmi+Ph4WSwWrVq1yt8hARcUyQBqpak8hhOoi9LSUvXo0UMLFy70dyiAX3BrIWqld+/euuqqq/Tss89Kql5FKyEhQffdd58eeughP0cH1B+LxaK33npLQ4cO9XcowAVDZQDnVfMYzpSUFNe+xvoYTgCA90gGcF4/9RjO/Px8P0UFAKgvJAMAAJgcyQDOqyk9hhMA4D2SAZxXU3oMJwDAezy1ELXSVB7DCdRFSUmJ9u3b53qdl5ennJwcRUdHq3379n6MDLgwuLUQtfbss8/qsccecz2Gc8GCBerdu7e/wwJ8tmnTJvXv3/+s/aNGjVJGRsaFDwi4wEgGAAAwOeYMAABgciQDAACYHMkAAAAmRzIAAIDJkQwAAGByJAMAAJgcyQAAACZHMgAAgMmRDAA+uvPOOzV06FDX6379+mnChAkXPI5NmzbJYrGosLDQ4xiLxaJVq1bV+pqzZs1Sz549fYrrq6++ksViUU5Ojk/XAdBwSAYQkO68805ZLBZZLBbZbDZ16tRJc+bMUVVVVYO/95tvvqm5c+fWamxtvsABoKHxoCIErIEDB+rll19WeXm53nnnHaWlpSkkJETTpk07a2xFRYVsNlu9vG90dHS9XAcALhQqAwhYoaGhiouLU2Jiou69916lpKTo7bfflvR9af+RRx5RfHy8OnfuLEk6dOiQfve736lFixaKjo7WTTfdpK+++sp1TYfDoUmTJqlFixZq1aqVpkyZoh8/3uPHbYLy8nJNnTpVCQkJCg0NVadOnfTiiy/qq6++cj0cp2XLlrJYLLrzzjslVT8iOj09XR06dFB4eLh69Oih119/3e193nnnHV166aUKDw9X//793eKsralTp+rSSy9VRESEOnbsqOnTp6uysvKscX/729+UkJCgiIgI/e53v1NRUZHb8SVLlqhr164KCwtTly5d9Nxzz3kdCwD/IRmAaYSHh6uiosL1esOGDcrNzVVmZqbWrFmjyspKpaamKjIyUlu3btUHH3yg5s2ba+DAga7znnjiCWVkZOill17Stm3bdPLkSb311ls/+b4jR47UP/7xDy1YsEB79uzR3/72NzVv3lwJCQl64403JEm5ubk6evSonn76aUlSenq6li5dqsWLF2v37t2aOHGibr/9dm3evFlSddIybNgwDRkyRDk5ORo7dqweeughr/9MIiMjlZGRoS+++EJPP/20XnjhBc2fP99tzL59+7Ry5UqtXr1a69at0yeffKI//vGPruPLli3TjBkz9Mgjj2jPnj169NFHNX36dL3yyitexwPATwwgAI0aNcq46aabDMMwDKfTaWRmZhqhoaHGgw8+6DoeGxtrlJeXu875+9//bnTu3NlwOp2ufeXl5UZ4eLixfv16wzAMo23btsa8efNcxysrK4127dq53sswDOPaa6817r//fsMwDCM3N9eQZGRmZp4zzn/961+GJOPUqVOufWVlZUZERISxfft2t7Fjxowxbr31VsMwDGPatGlGUlKS2/GpU6eeda0fk2S89dZbHo8/9thjRq9evVyvZ86caVitVuPw4cOufe+++64RFBRkHD161DAMw7jkkkuM5cuXu11n7ty5RnJysmEYhpGXl2dIMj755BOP7wvAv5gzgIC1Zs0aNW/eXJWVlXI6nbrttts0a9Ys1/Fu3bq5zRP49NNPtW/fPkVGRrpdp6ysTPv371dRUZGOHj2q3r17u44FBwfryiuvPKtVUCMnJ0dWq1XXXnttrePet2+fvv32W11//fVu+ysqKnT55ZdLkvbs2eMWhyQlJyfX+j1qrFixQgsWLND+/ftVUlKiqqoq2e12tzHt27fXRRdd5PY+TqdTubm5ioyM1P79+zVmzBiNGzfONaaqqkpRUVFexwPAP0gGELD69++vRYsWyWazKT4+XsHB7j/uzZo1c3tdUlKiXr16admyZWddq02bNnWKITw83OtzSkpKJElr1651+xKWqudB1JesrCyNGDFCs2fPVmpqqqKiovTqq6/qiSee8DrWF1544azkxGq11lusABoWyQACVrNmzdSpU6daj7/iiiu0YsUKxcTEnPXbcY22bdvqo48+Ut++fSVV/wacnZ2tK6644pzju3XrJqfTqc2bNyslJeWs4zWVCYfD4dqXlJSk0NBQHTx40GNFoWvXrq7JkDU+/PDD83/IH9i+fbsSExP15z//2bXvv//971njDh48qCNHjig+Pt71PkFBQercubNiY2MVHx+vAwcOaMSIEV69P4DGgwmEwHdGjBih1q1b66abbtLWrVuVl5enTZs26U9/+pMOHz4sSbr//vv117/+VatWrdLevXv1xz/+8SfXCLj44os1atQo3XXXXVq1apXrmitXrpQkJSYmymKxaM2aNTp+/LhKSkoUGRmpBx98UBMnTtQrr7yi/fv3a+fOnXrmmWdck/Luueceffnll5o8ebJyc3O1fPlyZWRkePV5f/azn+ngwYN69dVXtX//fi1YsOCckyHDwsI0atQoffrpp9q6dav+9Kc/6Xe/+53i4uIkSbNnz1Z6eroWLFig//znP9q1a5defvllPfnkk17FA8B/SAaA70RERGjLli1q3769hg0bpq5du2rMmDEqKytzVQoeeOAB3XHHHRo1apSSk5MVGRmp3/72tz953UWLFunmm2/WH//4R3Xp0kXjxo1TaWmpJOmiiy7S7Nmz9dBDDyk2Nlbjx4+XJM2dO1fTp09Xenq6unbtqoEDB2rt2rXq0KGDpOo+/htvvKFVq1apR48eWrx4sR599FGvPu+NN96oiRMnavz48erZs6e2b9+u6dOnnzWuU6dOGjZsmG644QYNGDBA3bt3d7t1cOzYsVqyZIlefvlldevWTddee60yMjJcsQJo/CyGp5lPAADAFKgMAABgciQDAACYHMkAAAAmRzIAAIDJkQwAAGByJAMAAJgcyQAAACZHMgAAgMmRDAAAYHIkAwAAmBzJAAAAJvf/AdJSuacKz29AAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cm = confusion_matrix(y_clean, y_pred_loo_our_model)\n", "disp = ConfusionMatrixDisplay(confusion_matrix=cm)\n", "disp.plot()\n", "print('{:<10} {:<15}'.format('Accuracy:', accuracy_score(y_clean, y_pred_loo_our_model)))\n", "print('{:<10} {:<15}'.format('Precision:', precision_score(y_clean, y_pred_loo_our_model)))\n", "print('{:<10} {:<15}'.format('Recall:', recall_score(y_clean, y_pred_loo_our_model)))\n", "print('{:<10} {:<15}'.format('F1:', f1_score(y_clean, y_pred_loo_our_model)))" ] }, { "cell_type": "markdown", "metadata": { "id": "uOnXu2eHHgUR" }, "source": [ "\n", "## Finding the best k in kNN 🔎\n", "\n", "What about the parameter k? How do we find the best k for our model? Why the VEGA model developers used k=4? Let's try to answer this...\n", "\n", "k is a hyperparameter of the model and should be chosen using a validation set.\n", "\n", "```{important}\n", "Remember to reserve your test set exclusively for assesing your model! Never use it for training or hyperparameter tuning! \n", "```" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 49, "referenced_widgets": [ "131bcbae8f1a4166b7c187e27752a4a1", "0d5dd3b8b7914bafb3c57066f2117f4b", "da3943a1533d48339cc99add56f296ec", "8ab9c22f3c6b407aab7c2b20542caddf", "62cc78dd0f354e4bab79b1adcbbb8d51", "bbb549238aca44c0bcc7b41a4e0516d4", "34b5cde70d7a41ae82eec617f48a33ff", "b01c55a604c741d3838d98d412f6098e", "99446b25af0a41efb78e8347a0468bc5", "536147b20e874c8a820f2ffbfd010b5a", "cbeab16ad6e344eea77e78270864e9bc" ] }, "id": "GLeXAFEJnQj8", "outputId": "59e721c0-50af-4ea9-a904-f99b613cb10e" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f04bf46177484c8b8837b8299e489551", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/50 [00:00" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.figure(figsize=(10,6))\n", "plt.plot(num_ks, train_accuracy, 'bs--', label='Train')\n", "plt.plot(num_ks, valid_accuracy, 'rx--', label='Validation')\n", "plt.xlabel('k')\n", "plt.ylabel('Misclasification rate')\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "kGhwzvOOHtd5" }, "source": [ "This graph is pretty similar to the one that we saw on slide 9 of Lecture 2. Here, we can see the expected general trend of the performance curves. \n", "\n", "Which k do you think is the best?" ] }, { "cell_type": "markdown", "metadata": { "id": "7VJZyifr-4nR" }, "source": [ "### Cross-validation\n", "\n", "Problem: if validation set is small and noisy, it might be misleading\n", "Idea: Increase the size of the validation set\n", "Problem: This would reduce the size of the training set\n", "\n", "Then, let's use all data for training and validation using k-fold cross-validation!" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "id": "t_6J0cmd_558" }, "outputs": [], "source": [ "from sklearn.model_selection import cross_validate" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 49, "referenced_widgets": [ "914eff8c2fae4f248af2dd1c0ab2adec", "c90eb39b1d6e4b5ba8136df7e5480a61", "b5249ba10245474e8ea18b502bebffb2", "3f07d346263c40ae80952f23251b4b74", "7976f19eb8f1414e8de608ed280755a7", "d23639c2608d4c4bbbb58fbeeff6c934", "164fc49ebc3b466f9b4c25260637210a", "4bfe0903581b4ca999017a19607380d5", "ef2b2ce93832431aac24fdb8dcbaf7f2", "d19e4dface4c4701a08ac745a31b8225", "24b693f1c368469faf63263a9c520158" ] }, "id": "jXOO3iN3-3n2", "outputId": "21856a82-4d55-49d0-eb6f-58e9a1645d5c" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "3e44fad5103941e3a431b8981714649f", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/49 [00:00" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.figure(figsize=(10,6))\n", "plt.plot(num_ks, train_misclassification, 'bs--', label='Train')\n", "plt.plot(num_ks, valid_misclassification, 'rx--', label='Validation')\n", "plt.xlabel('k')\n", "plt.ylabel('Misclasification rate')\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "1ruje72l3Gcv", "outputId": "f8955444-23e4-4014-83e5-f04032f69eeb" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "k with minimum validation misclassification: 17\n" ] } ], "source": [ "print('k with minimum validation misclassification: ', num_ks[np.argmin(valid_misclassification)])" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 388 }, "id": "LeGHGE0Z4Iw9", "outputId": "10e7209a-766e-4c65-ef65-7359fef9c92c" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA34AAAINCAYAAABh6K3PAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAACIhElEQVR4nO3dd3wU1frH8e8mIRQhoXdCEaQpQapYEVBERUBUxAICVyyIImJBpUhUUO9VULjYRb3SRLEj0hFEUYqiIAKigHSRQEIJ7M7vj/MbNgkJbJndzW4+79crr9mdnZlzkgxhnz3nPI/LsixLAAAAAICYFRfpDgAAAAAAQovADwAAAABiHIEfAAAAAMQ4Aj8AAAAAiHEEfgAAAAAQ4wj8AAAAACDGEfgBAAAAQIwj8AMAAACAGJcQ6Q7APx6PR9u3b1epUqXkcrki3R0AAAAAEWJZlg4ePKiqVasqLu7UY3oEflFm+/btqlGjRqS7AQAAAKCA2Lp1q6pXr37KYwj8okypUqUkmV9uUlJShHsDAAAAIFIOHDigGjVqnIgRToXAL8rY0zuTkpII/AAAAAD4tASM5C4AAAAAEOMI/AAAAAAgxhH4AQAAAECMY40fAAAAEEPcbreOHTsW6W7AIUWKFFF8fHzQ1yHwAwAAAGJERkaGtm3bJsuyIt0VOMTlcql69eoqWbJkUNch8AMAAABigNvt1rZt21SiRAlVqFDBp0yPKNgsy9KePXu0bds21atXL6iRPwI/AAAAIAYcO3ZMlmWpQoUKKl68eKS7A4dUqFBBf/zxh44dOxZU4EdyFwAAACCGMNIXW5z6fRL4AQAAAECMI/ADAAAAgBhH4AcAAAAgarVt21aDBg068bxWrVoaO3bsKc9xuVz66KOPgm7bqeuEA4EfAAAAAGnkSCktLe/X0tLM6w7r3Lmzrrjiijxf+/rrr+VyufTTTz/5dc3vv/9e/fv3d6J7J4wcOVJNmzY9af+OHTvUqVMnR9sKFQI/AAAAAFJ8vDR8+MnBX1qa2e9AEfHc+vXrpzlz5mjbtm0nvfbWW2+pRYsWatKkiV/XrFChgkqUKOFUF0+pcuXKKlq0aFjaChaBHwAAABDLMjPz/zpyxHvcsGHS44+bIG/YMPP6sGHm+eOPS0OG+HZdP1x99dWqUKGCJk2alGN/RkaG3n//fXXt2lU9e/ZUtWrVVKJECZ1zzjmaMmXKKa+Ze6rnhg0bdPHFF6tYsWJq1KiR5syZc9I5Dz/8sM466yyVKFFCderU0bBhw3Ts2DFJ0qRJk/TEE0/oxx9/lMvlksvlOtHf3FM916xZo3bt2ql48eIqV66c+vfvr4yMjBOv33bbberatav+/e9/q0qVKipXrpwGDBhwoq1QIvBD+ERg+gAAAEChV7Jk/l/du+c89vnnzfbJJ83rTz7pfZ57SmOtWnlf0w8JCQnq1auXJk2aJMuyTux///335Xa7dcstt6h58+b6/PPP9fPPP6t///669dZbtXz5cp+u7/F4dO211yoxMVHfffedXn75ZT388MMnHVeqVClNmjRJa9eu1bhx4/Taa6/phRdekCT16NFDDzzwgBo3bqwdO3Zox44d6tGjx0nXyMzMVMeOHVWmTBl9//33ev/99zV37lzdc889OY5bsGCBNm3apAULFujtt9/WpEmTTgp8Q4HAD+ETgekDAAAAKNj69u2rTZs2adGiRSf2vfXWW+revbtq1qypIUOGqGnTpqpTp44GDhyoK664QtOnT/fp2nPnztWvv/6qd955R6mpqbr44ov19NNPn3Tc448/rvPPP1+1atVS586dNWTIkBNtFC9eXCVLllRCQoIqV66sypUrq3jx4iddY/LkyTpy5IjeeecdnX322WrXrp3Gjx+vd999V7t27TpxXJkyZTR+/Hg1aNBAV199ta666irNmzfP3x+b3xJC3gJgGzbMbIcPN9MKuneXPv/cPB81yvs6AAAAnJNtquFJcn/wvnu3NGaMGeFLTJSyssw0z0cekeJyjRn98Ycj3WvQoIHOP/98vfnmm2rbtq02btyor7/+WqNGjZLb7dbTTz+t6dOn66+//lJWVpaOHj3q8xq+devWqUaNGqpateqJfW3atDnpuGnTpunFF1/Upk2blJGRoePHjyspKcmv72PdunVKTU3VGWeccWLfBRdcII/Ho/Xr16tSpUqSpMaNGys+28+9SpUqWrNmjV9tBYIRP4TXsGEmyHv6aal5c4I+AACAUDvjjPy/ihXLeezzz5ugb9Qo6ehRs33ySbM/9yhXftcMQL9+/fTBBx/o4MGDeuutt3TmmWfqkksu0XPPPadx48bp4Ycf1oIFC7R69Wp17NhRWVlZAf4wTrZs2TLdfPPNuvLKK/XZZ59p1apVeuyxxxxtI7siRYrkeO5yueTxeELSVnYEfgi/7EFefDxBHwAAQEFgL7/J/qG8/aF9Xst1HHTDDTcoLi5OkydP1jvvvKO+ffvK5XJp6dKl6tKli2655RalpqaqTp06+u2333y+bsOGDbV161bt2LHjxL5vv/02xzHffPONatasqccee0wtWrRQvXr19Oeff+Y4JjExUW63+7Rt/fjjj8rMluBm6dKliouLU/369X3uc6gQ+CH8sv/RcLtD+kcEAAAAPnK7856JZQd/pwl8glGyZEn16NFDQ4cO1Y4dO3TbbbdJkurVq6c5c+bom2++0bp163THHXfkWC93Oh06dNBZZ52l3r1768cff9TXX3+txx57LMcx9erV05YtWzR16lRt2rRJL774ombOnJnjmFq1amnz5s1avXq19u7dq6NHj57U1s0336xixYqpd+/e+vnnn7VgwQINHDhQt95664lpnpFE4Ifwsj9Jst16a8g/QQIAAIAPRo7MfybWsGEhz8Der18//fPPP+rYseOJNXmPP/64mjVrpo4dO6pt27aqXLmyunbt6vM14+LiNHPmTB0+fFitWrXSv/71Lz311FM5jrnmmmt0//3365577lHTpk31zTffaFiun0P37t11xRVX6NJLL1WFChXyLClRokQJzZ49W/v27VPLli113XXXqX379ho/frz/P4wQcFnZ86aiwDtw4ICSk5OVnp7u94LTiLODvkcfNWv8JJPk5dlnWesHAAAQpCNHjmjz5s2qXbu2iuVeu4eodarfqz+xAVk9ET729IF77jEZow4elIoW9QZ7IZw+AAAAABRmBH4In+zTA8aMkXbulLZulWrUYKQPAAAACCHW+CEyxo2Tzj7bBIAAAAAAQorAD+F36JBk1yrJlu4WAAAAQGgQ+CH83nlHsrMpEfgBAAA4ityNscWp3yeBH8Jv3z7v44yMyPUDAAAghsTHx0uSsrKyItwTOMn+fdq/30CR3AXh988/3seM+AEAADgiISFBJUqU0J49e1SkSBHFxTHGE+08Ho/27NmjEiVKKCEhuNCNwA/hl33Ej8APAADAES6XS1WqVNHmzZv1559/Rro7cEhcXJxSUlLkcrmCug6BH8Iv+4gfUz0BAAAck5iYqHr16jHdM4YkJiY6MnpL4Ifws0f8qlWT+vePbF8AAABiTFxcnIoVKxbpbqCAIfBD+Nkjfm++KV1+eWT7AgAAABQCBH4IvyuvlOrVk1JSIt0TAAAAoFAg8EP4jR4tWZa0dau0bp1Uv75E1ikAAAAgZAj8EBnHj0s1a5rH+/ZJZcpEtj8AAABADGOYBeHldksHDkgJCVKRImYfJR0AAACAkCLwQ3j99puUnCxVqiSdcYbZR+AHAAAAhBSBH8LLLuWQlETgBwAAAIQJgR/Cyy7lUKaMN/CjiDsAAAAQUiR3QXjZI35ly5r1fhIjfgAAAECIEfghvLKP+B0+bB4T+AEAAAAhReCH8Mo+4nf++VLz5lKdOpHtEwAAABDjCPwQXtkDv3vvjWxfAAAAgEKC5C4Ir6ZNpe7dpdTUSPcEAAAAKDRclmVZke4EfHfgwAElJycrPT1dSUlJke5OcDIzzZq/4sWlcuUi3RsAAAAgqvgTGzDih8h54gmpRg3p6acj3RMAAAAgphH4IbwyMyV7kJkC7gAAAEBYEPghvGrXlooUkX75RSpZ0uwj8AMAAABCisAP4WNZZk2f2y2VLu0d8cvIiGi3AAAAgFhH4IfwyciQjh83j8uUYaonAAAAECYEfggfu4Zf0aImkyeBHwAAABAWBH4In3/+MdsyZSSXi8APAAAACJOESHcAhYg94le2rNnWqiX162e2AAAAAEKGwA/hk33ET5Lq15defz1y/QEAAAAKCQI/hE/FilL37lKDBpHuCQAAAFCoEPghfC66yHzZLEs6eNCs8atc2az7AwAAAOA4Aj9EzsGDUnKyeXzokMn0CQAAAMBxZPVE+Bw9akb5bHZWT4nMngAAAEAIEfghfG65RSpSRHrtNfM8Pl4qVsw8JvADAAAAQobAD+Hzzz+S2y2VKOHdRy0/AAAAIOQI/BA+uev4SQR+AAAAQBgQ+CF8ctfxk6SSJc02IyP8/QEAAAAKCQI/hA8jfgAAAEBEUM4B4XH8uHTggHmcfcTv6quls8+WqlaNTL8AAACAQoDAD+Gxf7/3cfbAb/jwsHcFAAAAKGwI/BAeHo/Uvbt05IiUwG0HAAAAhBPvwBEeFStKM2acvN/jkQ4dkuLicpZ5AAAAAOAYkrsUAN26dVOZMmV03XXXRbor4TdokFSqlPT005HuCQAAABCzCPwKgPvuu0/vvPNOpLsRWsePS5Z18n47qyflHAAAAICQIfArANq2batSpUpFuhuh9corZm3fbbfl3E85BwAAACDkIh74TZw4UU2aNFFSUpKSkpLUpk0bzZo1y/FzArF48WJ17txZVatWlcvl0kcffXTSMRMmTFCtWrVUrFgxtW7dWsuXL3e8HzFh3z6zni8xMed+u4A7gR8AAAAQMhEP/KpXr64xY8ZoxYoV+uGHH9SuXTt16dJFv/zyi2PnLF26VMeOHTtp/9q1a7Vr165828nMzFRqaqomTJiQ5+vTpk3T4MGDNWLECK1cuVKpqanq2LGjdu/efeKYpk2b6uyzzz7pa/v27fm2G5P++cdssxdvlxjxAwAAAMIg4lk9O3funOP5U089pYkTJ+rbb79V48aNgz7H4/FowIABqlevnqZOnar4+HhJ0vr169WuXTsNHjxYDz30UJ7tdOrUSZ06dcq3788//7xuv/129enTR5L08ssv6/PPP9ebb76pRx55RJK0evXq/L/5wmTfPrPNXsNPIvADAAAAwiDiI37Zud1uTZ06VZmZmWrTpo0j58TFxemLL77QqlWr1KtXL3k8Hm3atEnt2rVT165d8w36TicrK0srVqxQhw4dcrTVoUMHLVu2LKBrnsqECRPUqFEjtWzZ0vFrh4Ud+OUe8WOqJwAAABByER/xk6Q1a9aoTZs2OnLkiEqWLKmZM2eqUaNGjp1TtWpVzZ8/XxdddJFuuukmLVu2TB06dNDEiRMD7vPevXvldrtVqVKlHPsrVaqkX3/91a9rdejQQT/++KMyMzNVvXp1vf/++ycFsQMGDNCAAQN04MABJScnB9zviLGneuYe8UtJMYXdGzQIf58AAACAQqJABH7169fX6tWrlZ6erhkzZqh3795atGjRKYM/f89JSUnRu+++q0suuUR16tTRG2+8IZfLFapvyS9z586NdBdCL78Rv6ZN8y7sDgAAAMAxBWKqZ2JiourWravmzZtr9OjRSk1N1bhx4xw9Z9euXerfv786d+6sQ4cO6f777w+qz+XLl1d8fPxJyWF27dqlypUrB3XtmHThhVK7dlLVqpHuCQAAAFDoFIjALzePx6OjR486ds7evXvVvn17NWzYUB9++KHmzZunadOmaciQIQH3MTExUc2bN9e8efNy9GHevHk+r08sVF55RZo3L+8pnZYlHToU/j4BAAAAhUTEp3oOHTpUnTp1UkpKig4ePKjJkydr4cKFmj17tiRp/PjxmjlzZo4A63TnZOfxeNSpUyfVrFlT06ZNU0JCgho1aqQ5c+aoXbt2qlatWr6jfxkZGdq4ceOJ55s3b9bq1atVtmxZpaSkaPDgwerdu7datGihVq1aaezYscrMzDyR5RM+2L1bqlzZBH9utxRXID+LAAAAAKJaxAO/3bt3q1evXtqxY4eSk5PVpEkTzZ49W5dddpkkM1q3adMmv87JLi4uTk8//bQuuugiJWYrHp6amqq5c+eqQoUK+fbthx9+0KWXXnri+eDBgyVJvXv31qRJk9SjRw/t2bNHw4cP186dO9W0aVN9+eWXJyV8KfQsy2zzWlN5xhne1w8f9pZ3AAAAAOAYl2XZ77oRDeysnunp6UpKSop0d3yzZo1J4lKvnpQ746nHI/1/bUXt3CkRNAMAAAA+8Sc2YF4dQm/fPhPg5SUuTipRwjymlh8AAAAQEgR+CL38avjZKOIOAAAAhBSBH0Ivvxp+NntdX0ZGePoDAAAAFDIEfgi904342YEfI34AAABASEQ8qycKgdON+LVvbxK/5BcYAgAAAAgKgR9C73QjfmPHhq0rAAAAQGHEVE+EXr16Urt2UoMGke4JAAAAUChRxy/KRGUdP19YVs6afgAAAABOiTp+iC4DB0oJCdLo0ZHuCQAAABCTCPwQeqcbVI6PN6N9ZPUEAAAAQoLAD6FXqZJUrpy0cWPer1PAHQAAAAgpsnoitDweae9eM+pnB3i5UcAdAAAACClG/BBa6eneqZ4UcAcAAAAigsAPoWXX8DvjDKlo0byPYaonAAAAEFIEfgitffvMNr/RPompngAAAECIscYPoWUHfmXL5n9M9epShw5Samp4+gQAAAAUMgR+CC17quepRvwuuECaMyc8/QEAAAAKIQI/hFbp0lK7dtK550a6JwAAAEChReCH0OrY0XwBAAAAiBiSuyDytm0zU0GTkiLdEwAAACAmMeKHyCtWTNq/3zx2u6X4+Ih2BwAAAIg1jPghtK6/XipXTnrvvfyPscs5SNTyAwAAAEKAwA+htWePKemQcIrB5WLFpLj/vxUJ/AAAAADHEfghtHyp4+dyUcQdAAAACCECP4SWL3X8JG/gx4gfAAAA4DgCP4SWLyN+EoEfAAAAEEJk9UToHD0qHTpkHp9uxK9NGyklRSpePPT9AgAAAAoZAj+Ejj3N0+WSkpNPfey774a+PwAAAEAhReCH0HG7pfbtpePHvVk7AQAAAIQdgR9Cp1o1ae7cSPcCAAAAKPQYhkHBMGiQKfQ+blykewIAAADEHAI/FAxHjpgMoOnpke4JAAAAEHMI/BA648ebUbz77jv9sSVLmi0F3AEAAADHEfghdPbsMaN4x46d/ljq+AEAAAAhQ+CH0LHLOZyuhp9E4AcAAACEEIEfQmffPrMtW/b0x9qBH1M9AQAAAMcR+CF0GPEDAAAACgQCP4SOPyN+VatKLVpI9eqFtk8AAABAIUQBd4SOP4Hf5ZebLwAAAACOI/BD6DRpYqZwVqoU6Z4AAAAAhRqBH0Ln/fcj3QMAAAAAYo0fCoo//pBq15bq1o10TwAAAICYw4gfCoaEBBP8JXBLAgAAAE5jxA+hsXq1VK6cdMEFvh1vl3M4flzKygpZtwAAAIDCiMAPofH33yar5/79vh1vB34StfwAAAAAhxH4ITTs4u2+lHKQpMREqUgR85jADwAAAHAUgR9Cw67hV6aM7+fYo34ZGc73BwAAACjECPwQGv6O+EnewI8RPwAAAMBRpFBEaAQy4nfOOabYO5k9AQAAAEfxDhuhYY/4+RP4zZoVmr4AAAAAhRxTPREa1atLzZpJtWpFuicAAABAoeeyLMuKdCfguwMHDig5OVnp6elKSkqKdHcAAAAARIg/sQEjfig4hgyRzjxTeuutSPcEAAAAiCkEfig49u6Vfv9d2r070j0BAAAAYgqBH0Kjdm2pXj1p61bfz6GcAwAAABASZPWE844dk/74wzwuXtz380qWNFsCPwAAAMBRjPjBefv3ex+XLu37eYz4AQAAACFB4Afn2TX8kpL8K8ZO4AcAAACEBIEfnLdvn9mWLevfeQR+AAAAQEgQ+MF59oifv4FfxYpS3bpSpUrO9wkAAAAoxEjuAufZI35lyvh33rXXmi8AAAAAjmLED8474wypWTOpYcNI9wQAAACAGPFDKHTtar4AAAAAFAiM+KHg2LBBatpUOv/8SPcEAAAAiCmM+KHgcLmkH3+USpWKdE8AAACAmMKIH5x3yy0mO+fMmf6dZ5dzyMiQLMv5fgEAAACFFIEfnPfHH9KmTZLH4995duBnWdKRI453CwAAACisCPzgvGALuEsUcQcAAAAcROAH59kF3P2t4xcfLxUrZh5nZDjbJwAAAKAQI/CDsywr8BE/yTvqx4gfAAAA4BiyesJZhw9LWVnmsb8jfpJUu7aUnOz/+kAAAAAA+SLwg7Ps0b6EBKlkSf/P//57Z/sDAAAAgMAPDjt2TGrWTIqLM3X5AAAAAEQcgR+cVbu2tGJFpHsBAAAAIBuSu6BgefhhM2L4wQeR7gkAAAAQMwj8ULD88Ye0apX011+R7gkAAAAQMwj84Kz//leqW1caPjyw8+2EMJRzAAAAABxD4Adn/fWXtGmTtH9/YOfbdfwo4A4AAAA4JqDA791339UFF1ygqlWr6s8//5QkjR07Vh9//LGjnUMU+ucfsw2khp9EAXcAAAAgBPwO/CZOnKjBgwfryiuv1P79++V2uyVJpUuX1tixY53uH6KNXcevbNnAzifwAwAAABznd+D30ksv6bXXXtNjjz2m+Pj4E/tbtGihNWvWONo5RKFgR/xY4wcAAAA4zu/Ab/PmzTr33HNP2l+0aFFl8mYdwY74lSkjVajgHfkDAAAAEDS/A7/atWtr9erVJ+3/8ssv1bBhQyf6hGhmj/gFGvj17i3t3i299ppzfQIAAAAKuQR/Txg8eLAGDBigI0eOyLIsLV++XFOmTNHo0aP1+uuvh6KPiCY1a0oej1S+fKR7AgAAAOD/+R34/etf/1Lx4sX1+OOP69ChQ7rppptUtWpVjRs3TjfeeGMo+ohoMm9epHsAAAAAIJeAyjncfPPN2rBhgzIyMrRz505t27ZN/fr1c7pvhUK3bt1UpkwZXXfddZHuSsHw66/SJZdIXbpEuicAAABAzPA78GvXrp32/39x7hIlSqhixYqSpAMHDqhdu3aOdq4wuO+++/TOO+9EuhsFx7Fj0uLF0rJlke4JAAAAEDP8DvwWLlyorKysk/YfOXJEX3/9tSOdKkzatm2rUqVKRbobzli9WjrzTOnqqwO/BnX8AAAAAMf5HPj99NNP+umnnyRJa9euPfH8p59+0qpVq/TGG2+oWrVqfndg4sSJatKkiZKSkpSUlKQ2bdpo1qxZ+R4/evRotWzZUqVKlVLFihXVtWtXrV+/3u92T2fx4sXq3LmzqlatKpfLpY8++ijP4yZMmKBatWqpWLFiat26tZYvX+54X6LG7t3S779LW7YEfg27jt+hQyZJDAAAAICg+ZzcpWnTpnK5XHK5XHlO6SxevLheeuklvztQvXp1jRkzRvXq1ZNlWXr77bfVpUsXrVq1So0bNz7p+EWLFmnAgAFq2bKljh8/rkcffVSXX3651q5dqzPyqf22dOlStWrVSkWKFMmxf+3atSpXrpwqVap00jmZmZlKTU1V3759de211+Z53WnTpmnw4MF6+eWX1bp1a40dO1YdO3bU+vXrT0yBbdq0qY4fP37SuV999ZWqVq162p9PVAm2hp+Us37foUPeQBAAAABAwFyWZVm+HPjnn3/KsizVqVNHy5cvV4UKFU68lpiYqIoVKyo+Pt6RTpUtW1bPPfecTwlj9uzZo4oVK2rRokW6+OKLT3rd4/GoWbNmqlevnqZOnXqij+vXr9cll1yiwYMH66GHHjplGy6XSzNnzlTXrl1z7G/durVatmyp8ePHn2irRo0aGjhwoB555BEfv1szfXb8+PGaMWPGaY89cOCAkpOTlZ6erqSkJJ/bCIuJE6W775a6dZM+/DCwa3g8kn0f7dwp5RGUAwAAAPAvNvB5xK9mzZqSTHATKm63W++//74yMzPVpk0bn85JT0+XZILFvMTFxemLL77QxRdfrF69eundd9/V5s2b1a5dO3Xt2vW0QV9+srKytGLFCg0dOjRHWx06dNCyECQmmTBhgiZMmCC32+34tR1jj/iVKRP4NeLipBIlzGgf6/wAAAAAR/hdx8+2du1abdmy5aREL9dcc43f11qzZo3atGmjI0eOqGTJkpo5c6YaNWp02vM8Ho8GDRqkCy64QGeffXa+x1WtWlXz58/XRRddpJtuuknLli1Thw4dNHHiRL/7atu7d6/cbvdJ00QrVaqkX3/91efrdOjQQT/++KMyMzNVvXp1vf/++3kGvQMGDNCAAQNORPUF0j//mG0wUz0lU/w9M1PKI4kQAAAAAP/5Hfj9/vvv6tatm9asWSOXyyV7pqjL5ZKkgEak6tevr9WrVys9PV0zZsxQ7969tWjRotMGfwMGDNDPP/+sJUuWnLaNlJQUvfvuu7rkkktUp04dvfHGGyf6HElz586NdBecYwd+wYz4SdKffwbfFwAAAAAn+F3O4b777lPt2rW1e/dulShRQr/88osWL16sFi1aaOHChQF1IjExUXXr1lXz5s01evRopaamaty4cac855577tFnn32mBQsWqHr16qdtY9euXerfv786d+6sQ4cO6f777w+or7by5csrPj5eu3btOqmdypUrB3XtqFWmjFS7tlSlSqR7AgAAACAbvwO/ZcuWadSoUSpfvrzi4uIUFxenCy+8UKNHj9a9997rSKc8Ho+OHj2a52uWZemee+7RzJkzNX/+fNWuXfu019u7d6/at2+vhg0b6sMPP9S8efM0bdo0DRkyJOA+JiYmqnnz5po3b16Ofs+bN8/n9Ykx59//NuUc+vSJdE8AAAAAZON34Od2u08UHC9fvry2b98uySR/CaSe3tChQ7V48WL98ccfWrNmjYYOHaqFCxfq5ptvliSNHz9e7du3P3H8gAED9L///U+TJ09WqVKltHPnTu3cuVOHDx/O8/oej0edOnVSzZo1NW3aNCUkJKhRo0aaM2eO3nrrLb3wwgt5npeRkaHVq1dr9erVkqTNmzdr9erV2pKtRt3gwYP12muv6e2339a6det01113KTMzU30IfIIzbJh06aXS7NmR7gkAAAAQE/xe43f22Wfrxx9/VO3atdW6dWs9++yzSkxM1Kuvvqo6der43YHdu3erV69e2rFjh5KTk9WkSRPNnj1bl112mSQzWrdp06YTx9sJWdq2bZvjOm+99ZZuu+22k64fFxenp59+WhdddJESExNP7E9NTdXcuXNzlKXI7ocfftCll1564vngwYMlSb1799akSZMkST169NCePXs0fPhw7dy5U02bNtWXX36ZZ11A+OGnn6SFC6Wbbop0TwAAAICY4HMdP9vs2bOVmZmpa6+9Vhs3btTVV1+t3377TeXKldO0adPyLO4O5xToOn5nny0VLy59+qkUzDrHm2+WJk+Wnn9eCnItJgAAABCrQlLHz9axY8cTj+vWratff/1V+/btU5kyZQpElkxEyJEj0i+/mMfFigV3rTPOMFvq+AEAAACO8GuN37Fjx5SQkKCff/45x/6yZcsS9BV2dimHuDgp2JFIAj8AAADAUX4FfkWKFFFKSkpAtfoQ4+zAr3RpE/wFg8APAAAAcJTf79Afe+wxPfroo9q3b18o+oNoZd8PZcsGfy078MvICP5aAAAAAPxf4zd+/Hht3LhRVatWVc2aNXWG/Sb9/61cudKxziGK2CN+ZcoEf60zzpCyZWAFAAAAEBy/A7+uXbuGoBuIek6O+A0cKN17b/DXAQAAACApgMBvxIgRoegHol2RIlLt2lKNGsFfi0RBAAAAgKP8ruOHyCrQdfwAAAAAhI0/sUGQ6ReBEFi3TurSRerbN9I9AQAAAGKC31M9gZDLyJA++cSZaaMAAAAAGPGDQ/r1k1q0kGbPDv5a1PEDAAAAHEXgB2f88ou0YoV05Ejw1yLwAwAAABzl91RPt9utSZMmad68edq9e7c8Hk+O1+fPn+9Y5xBFnK7jJ0lHj0rHj0sJzEgGAAAAguH3O+r77rtPkyZN0lVXXaWzzz5bLlLvQ3K2jl/Jkt7HmZlScnLw1wQAAAAKMb8Dv6lTp2r69Om68sorQ9EfRCPL8o74ORH4FS0qxcVJHg+BHwAAAOAAv9f4JSYmqm7duqHoC6LVwYOS220eOzHV0+Uy0z3j46XDh4O/HgAAAFDI+R34PfDAAxo3bpyo+44T7GmexYpJxYs7c829e6Vjx6Qzz3TmegAAAEAh5vdUzyVLlmjBggWaNWuWGjdurCJFiuR4/cMPP3Ssc4gSR49KdepIiYnOXdPJawEAAACFnN+BX+nSpdWtW7dQ9AXRqn59adOmSPcCAAAAQD78DvzeeuutUPQDyCktTVq5UnrgAenCCyPdGwAAACCqBVzAfc+ePVqyZImWLFmiPXv2ONknQFqyRProI2nz5uCvNXKkCSTzkpZmXgcAAABimN+BX2Zmpvr27asqVaro4osv1sUXX6yqVauqX79+OnToUCj6iILu1VelFi2k555z7pp2EfeMjOCvFR8vDR9+cvCXlmb2x8cH3wYAAABQgPkd+A0ePFiLFi3Sp59+qv3792v//v36+OOPtWjRIj3wwAOh6CMKuk2bpBUrpB07nLumHfhlZgZ/rWHDpFGjcgZ/dtA3apR5HQAAAIhhfq/x++CDDzRjxgy1bdv2xL4rr7xSxYsX1w033KCJEyc62T9EA7t4uxM1/GwlS5qtE4GfZIK7I0dMsPfkk1JWFkEfAAAACg2/R/wOHTqkSpUqnbS/YsWKTPUsrOw6fmXLOndNJ0f8bA0amG1WlikXQdAHAACAQsLvwK9NmzYaMWKEjhw5cmLf4cOH9cQTT6hNmzaOdg5RIhQjfqEI/D77zGxdLhP85ZfwBQAAAIgxfk/1HDdunDp27Kjq1asrNTVVkvTjjz+qWLFimj17tuMdRBQIxYif01M909Kk6dPN4xo1pH/9y0z7lBj5AwAAQMzzO/A7++yztWHDBr333nv69ddfJUk9e/bUzTffrOLFizveQUSBUIz4DRwo3XOPVKxY8NeyE7n06ye98Ya0bZv06KPmNYI/AAAAFAJ+B36SVKJECd1+++1O9wXRqnRpKT3d2RE/JwI+m9ttErm89ZZ57vGYDKR2sOd2O9cWAAAAUAC5LMuyTnfQJ598ok6dOqlIkSL65JNPTnnsNddc41jncLIDBw4oOTlZ6enpSkpKinR3oodlmemjdgKipUul88+PbJ8AAACAIPgTG/g04te1a1ft3LlTFStWVNeuXfM9zuVyyc3oCZzwyy/SM89IlSo5Uxj+wAFv0CdJW7cGf00AAAAgSviU1dPj8ahixYonHuf3RdAHx+zbJ737rnSaEWafbd/ufRwf701IAwAAABQCfpdzeOedd3T06NGT9mdlZemdd95xpFOIIqtXSy1aSL16OXtdp8s57NhhttWqmULud93lzHUBAACAKOB34NenTx+lp6eftP/gwYPq06ePI51CFNmxQ1qxQvr5Z2ev63TgZ4/41a8vJQSU0wgAAACIWn4HfpZlyeVynbR/27ZtSk5OdqRTiCKhqOEneQO/jAxnrmcHflWrOnM9AAAAIIr4PPRx7rnnyuVyyeVyqX379krINmridru1efNmXXHFFSHpJAqwUNTwk7wF3I8fl7KypMTE4K537rnSgAFSzZrSTTeZZC+ffRZ8PwEAAIAo4HPgZ2fzXL16tTp27KiS9htzSYmJiapVq5a6d+/ueAdRwIV6xE8y0z2DDfwuu8x87dsnlStn9h0+LBUvHtx1AQAAgCjgc+A3YsQISVKtWrXUo0cPFXOywDailz3i53TgV6SI+Tp2zAR+To0olikjlShhSjts2ybVq+fMdQEAAIACzO81fr179ybog5c94uf0VE/JBGYZGSYTZ7DWr5f+/ts8rlHDbKnlBwAAgELC7/SGbrdbL7zwgqZPn64tW7YoKysrx+v7qI9WuCQmmqDPnj7ppP+vHRk0yzJr/A4fljZulFJSTCBI4AcAAIBCwu8RvyeeeELPP/+8evToofT0dA0ePFjXXnut4uLiNHLkyBB0EQXaa6+ZUb9+/SLdk/ylp5ugT5KqVPGO+G3ZErk+AQAAAGHkd+D33nvv6bXXXtMDDzyghIQE9ezZU6+//rqGDx+ub7/9NhR9RGH17LNSnz7SqlXBXccu3l66tFnfx1RPAAAAFDJ+B347d+7UOeecI0kqWbLkiWLuV199tT7//HNne4fC7bPPpEmTzPTMYOSu4ZeSIsXHe0cBAQAAgBjnd+BXvXp17fj/EZQzzzxTX331lSTp+++/V9GiRZ3tHQo2y5LatDFlEuzEKU6ySzpkZgZ3ndyB3y23SEeOSO++G9x1AQAAgCjhd3KXbt26ad68eWrdurUGDhyoW265RW+88Ya2bNmi+++/PxR9REF16JBkT+8NRdBv14oMNvCzp3ragV+wNQEBAACAKON34DdmzJgTj3v06KGaNWvqm2++Ub169dS5c2dHO4cCzs7gWqRIzoLrTrGvmZER3HXsEb8qVYK7DgAAABCl/A78cjvvvPN03nnnOdEXRBu7eHuZMpLL5fz1nZrq2bat5PFIF1/s3XfXXdKaNdLrr0sNGgR3fQAAAKCA8zvwGz16tCpVqqS+ffvm2P/mm29qz549evjhhx3rHAqokSNNcpSLLjLPy5b1vpaWJrnd5phgOTXVs2tX85Xdt99Kq1dLmzYR+AEAACDm+Z3c5ZVXXlGDPN4oN27cWC+//LIjnUIBFx8vDR8uvfKKeV6mjNmmpZn98fHOtOPUVM+8pKSYLSUdAAAAUAj4PeK3c+dOVcljrVSFChVOZPtEjBs2zGyHDzfbsmW9Qd+oUd7Xg3XvvdLtt0vJyYFfw7LMlM4qVaTy5b1TUiniDgAAgELE7xG/GjVqaOnSpSftX7p0qaraWRMR+4YNk665xjyeNcv5oE8yBderVDFF1wO1f7+UmipVrCgdPerdz4gfAAAAChG/R/xuv/12DRo0SMeOHVO7du0kSfPmzdNDDz2kBx54wPEOogD7+GNTxiEry5RIcDLoc4qd0bNMGalYMe9+RvwAAABQiPgd+D344IP6+++/dffddysrK0uSVKxYMT388MMaOnSo4x1EAZaW5g36srLMcyeDv19+MesIq1SRAr23chdvt9mBHyN+AAAAKAT8nurpcrn0zDPPaM+ePfr222/1448/at++fRpur/dC4ZB9Td/Ro2Y7fLjZ75S//pJeekmaOjXwa+Qu3m5LSTFJaOLjzTpAAAAAIIYFXMevZMmSatmypZN9QbTIK5FL7oQvToz8OVHHL7/i7TVqmIDVqQykAAAAQAHmU+B37bXXatKkSUpKStK11157ymM//PBDRzqGAsztzjuRi/3c7XamHScDv9wjfi4XQR8AAAAKDZ8Cv+TkZLn+Pw1+UlLSiccopE5VnN3JNX5OFHDPb6onAAAAUIj4FPh169ZNxf4/I+KkSZNC2R/AK3sBd8vy1uDzR5cuZppnixYnv/bii9L06VLfvuYLAAAAiFE+JXfp1q2b9u/fL0mKj4/X7t27Q9knwLADP8uSjhwJ7Bq33GICvDZtTn5t61Zp6VLp558D7yMAAAAQBXwK/CpUqKBvv/1WkmRZFlM9ER524CcFN90zP5R0AAAAQCHh01TPO++8U126dJHL5ZLL5VLlypXzPdbtVGIPID5e+u03qUQJU4DdX0ePSuvXm6meFSqc/HpKitlSxB0AAAAxzqfAb+TIkbrxxhu1ceNGXXPNNXrrrbdUunTpEHcNkFSvXuDnbtggpaZK5cpJe/ee/DojfgAAACgkfK7j16BBAzVo0EAjRozQ9ddfrxIlSoSyX0DwTpfR0x7x27lTysqSEhPD0y8AAAAgzHxa45fdiBEjCPoQPi++KA0aJP36q//n5le83Va+vFSsmEke89dfAXcRAAAAKOh8GvFr1qyZ5s2bpzJlyujcc889ZXKXlStXOtY5QO++K/3wg9Shg9SggX/n5le83eZySbVrm3IR6enB9RMAAAAowHwK/Lp06aKiRYtKkrp27RrK/gA5BVPE3Zfi7b/8Elh9QAAAACCK+BT4jRgxIs/HQMhlL+Lur9NN9ZQI+gAAAFAo+L3Gb+vWrdq2bduJ58uXL9egQYP06quvOtoxQJI38AtkxO90Uz0BAACAQsLvwO+mm27SggULJEk7d+5Uhw4dtHz5cj322GMaNWqU4x1EIRfMVM/evaV775UaN87/mPnzpQsukPr2Dax/AAAAQBTwuZyD7eeff1arVq0kSdOnT9c555yjpUuX6quvvtKdd96p4cOHO95JFGLBTPW8447TH5OVJX3zjXTwoP/XBwAAAKKE3yN+x44dO5HoZe7cubrmmmskmTp/O+xkGoBTgpnq6Qu7lh9F3AEAABDD/A78GjdurJdffllff/215syZoyuuuEKStH37dpUrV87xDqKQu+8+ad06adgw/847cEBavVras+fUx9WoYbb79zPqBwAAgJjld+D3zDPP6JVXXlHbtm3Vs2dPpaamSpI++eSTE1NAAcdUrmzq9/n7ocKyZdK550rt25/6uFKlpORk85hRPwAAAMQov9f4tW3bVnv37tWBAwdUpkyZE/v79++vEiVKONo5IGD+ZPRMSZHWrDGBX6NGoe0XAAAAEAF+j/hJUnx8fI6gT5Jq1aqlihUrOtIp4IR166Thw6WJE/07z5fi7TZ7uueWLf61AQAAAEQJv0f8JGnGjBmaPn26tmzZoqysrByvrVy50pGOAZKkDRuktDSpdWvprrt8P8+X4u22OnW8SV4AAACAGOT3iN+LL76oPn36qFKlSlq1apVatWqlcuXK6ffff1enTp1C0UcUZoFm9fRnqudLL0l//indfrt/bQAAAABRwu/A77///a9effVVvfTSS0pMTNRDDz2kOXPm6N5771V6enoo+ojCLNDAz5+pngAAAECM8zvw27Jli84//3xJUvHixXXw/1Pg33rrrZoyZYqzvQMCLeDuz1RPAAAAIMb5HfhVrlxZ+/btkySlpKTo22+/lSRt3rxZlmU52zugZEmz9XfE7777zFedOqc/ds8e6YILpHr1JO5hAAAAxCC/k7u0a9dOn3zyic4991z16dNH999/v2bMmKEffvhB1157bSj6iMLMHvE7dEjyeKQ4Hz+rGDzY9zaSkqRvvjGP9+6VKlTwr48AAABAAeey/Bym83g88ng8SkgwMePUqVP1zTffqF69errjjjuUmJgYko7COHDggJKTk5Wenq6kpKRIdyf0MjO9o34HD3ofO61KFWnnTmnFCqlZs9C0AQAAADjIn9jA7xG/uLg4xWUbdbnxxht14403+t9LwBclSkjLl5uRv+LFfTtnzx7pr7+k6tWl8uV9O6dGDRP4bdlC4AcAAICY41Pg99NPP/l8wSZNmgTcGeAkLpfUsqV/53z2mdS3r9Spk/TFF76dk5Iiff+9tHWr/30EAAAACjifAr+mTZvK5XKdNnmLy+WS2+12pGNAwALJ6FmjhtkS+AEAACAG+RT4bd68OdT9APL3+uvStm3SbbdJtWqd/nh/irfbUlLMdssWf3sHAAAAFHg+BX41a9YMdT+A/L34orRmjXThhaEL/GrXNsFfmTIBdREAAAAoyPyu4zd69Gi9+eabJ+1/88039cwzzzjSKSAHf4u4BzLVs2tX6c8/pYkT/eoaAAAAEA38DvxeeeUVNWjQ4KT9jRs31ssvv+xIpwqbbt26qUyZMrruuusi3ZWCyQ78fC3ivmOH2foz4gcAAADEML8Dv507d6pKHiMpFSpU0A77DTf8ct999+mdd96JdDcKLn8CP4+HwA8AAADIxe/Ar0aNGlq6dOlJ+5cuXaqqvNEOSNu2bVWqVKlId6Pg8ifwO35cevppadAgqVIl/9rp0cOs81u2zO8uAgAAAAWZ34Hf7bffrkGDBumtt97Sn3/+qT///FNvvvmm7r//ft1+++1+d2DixIlq0qSJkpKSlJSUpDZt2mjWrFmnPGfx4sXq3LmzqlatKpfLpY8++sjvdn3hSzsTJkxQrVq1VKxYMbVu3VrLly8PSV8KNX8Cv8RE6cEHpRdekIoU8a+dnTtNOYc//vC7iwAAAEBB5lNWz+wefPBB/f3337r77ruVlZUlSSpWrJgefvhhDR061O8OVK9eXWPGjFG9evVkWZbefvttdenSRatWrVLjxo3zPCczM1Opqanq27evrr322tO2sXTpUrVq1UpFcgUCa9euVbly5VQpn5Gh07Uzbdo0DR48WC+//LJat26tsWPHqmPHjlq/fr0qVqwoydRAPH78+EnnfvXVV4yQ+qpkSbP1dY1foKjlBwAAgBjlsk5XlT0fGRkZWrdunYoXL6569eqpaNGijnWqbNmyeu6559SvX7/THutyuTRz5kx17do1z9c9Ho+aNWumevXqaerUqYqPj5ckrV+/XpdccokGDx6shx56KKB2WrdurZYtW2r8+PEn2qpRo4YGDhyoRx555PTfaDYLFy7U+PHjNWPGjFMed+DAASUnJys9PV1JSUl+tRG1fv9d2r3bBGbVqp362K1bpb17pZo1pbJl/Wvn0Uel0aOle+6RXnop8P4CAAAAYeBPbOD3VE9byZIl1bJlS6WkpGjWrFlat25doJc6we12a+rUqcrMzFSbNm2Cvp4kxcXF6YsvvtCqVavUq1cveTwebdq0Se3atVPXrl19CvrykpWVpRUrVqhDhw452urQoYOWhWCN2IQJE9SoUSO1bNnS8WsXeHXqSOedd/qgT5ImTZKaNZMC+b3aI34UcQcAAECM8Tvwu+GGG06McB0+fFgtWrTQDTfcoCZNmuiDDz4IqBNr1qxRyZIlVbRoUd15552aOXOmGjVqFNC18lK1alXNnz9fS5Ys0U033aR27dqpQ4cOmhhEzba9e/fK7XafNE20UqVK2rlzp1/X6tChg66//np98cUXql69ep6B44ABA7R27Vp9//33Afe5UAgmo2dKitky1RMAAAAxxu/Ab/HixbroooskSTNnzpRlWdq/f79efPFFPfnkkwF1on79+lq9erW+++473XXXXerdu7fWrl0b0LXyk5KSonfffVfTpk1TQkKC3njjDblcLkfbCNTcuXO1Z88eHTp0SNu2bXNstDNm/Pqr9J//SFOmnP7YQIq32xjxAwAAQIzyO/BLT09X2f9fO/Xll1+qe/fuKlGihK666ipt2LAhoE4kJiaqbt26at68uUaPHq3U1FSNGzcuoGvlZ9euXerfv786d+6sQ4cO6f777w/qeuXLl1d8fLx27dp1UjuVK1cO6trI5ccfpSFDpFdeOf2xduAX6IhfSorUsKEpCwEAAADEiIDq+C1btkyZmZn68ssvdfnll0uS/vnnHxUrVsyRTnk8Hh09etSRa0lmWmb79u3VsGFDffjhh5o3b56mTZumIUOGBHzNxMRENW/eXPPmzTuxz+PxaN68eYzYOc2frJ7BTPUsXVr680/p66+lBL8T3gIAAAAFlt/vbgcNGqSbb75ZJUuWVM2aNdW2bVtJZgroOeec43cHhg4dqk6dOiklJUUHDx7U5MmTtXDhQs2ePVuSNH78eM2cOTNHgJWRkaGNGzeeeL5582atXr1aZcuWVYq9Tuv/eTwederUSTVr1jwxzbNRo0aaM2eO2rVrp2rVquU7+ne6dgYPHqzevXurRYsWatWqlcaOHavMzEz16dPH758DTsHXOn4ejzfwC2SqJwAAABCj/A787r77brVq1Upbt27VZZddprg4M2hYp06dgNb47d69W7169dKOHTuUnJysJk2aaPbs2brsssskmdG6TZs25Tjnhx9+0KWXXnri+eDBgyVJvXv31qRJk3IcGxcXp6effloXXXSREhMTT+xPTU3V3LlzVaFChXz7drp2evTooT179mj48OHauXOnmjZtqi+//DLfuoAIkB34ZWSc+rg9eyS3W3K5JH4HAAAAwAkB1/FDZBTKOn7r1kmNGpm6fH//nf9x+/dLr70mpadLASYa0jPPSBMmSHfcIT32WGDXAAAAAMLAn9jApxG/wYMHKy0tTWecccaJUa/8PP/88773FPCFr1M9S5eWHnwwuLayskw5h82bg7sOAAAAUID4FPitWrVKx44dO/E4PwWlPAJijB34HT1qpnLGx4euLbukA7X8AAAAEEN8CvwWLFiQ52MgLJKTpa++MgHgqT5c2LhROnBAql1bKlMmsLbs5EDU8gMAAEAM8bucAxB2CQnSZZdJ558vxZ3ilv3Pf6TmzaWxYwNvK/uIH8tfAQAAECN8zurZt29fn4578803A+4MEJRgirfbqlc328xM6Z9/TEIZAAAAIMr5HPhNmjRJNWvW1LnnnisSgSLsJk+Wdu2SevaUKlfO+xgnAr/ixaUKFUxpiK1bCfwAAAAQE3wO/O666y5NmTJFmzdvVp8+fXTLLbeoLG+KES4jRpg1fC1bnj7wC7Z4+3nnmdE+tzu46wAAAAAFhM9r/CZMmKAdO3booYce0qeffqoaNWrohhtu0OzZsxkBROidrqSD221GBKXgRvwk6ZNPpK+/lpo1C+46AAAAQAHhV3KXokWLqmfPnpozZ47Wrl2rxo0b6+6771atWrWUkZERqj4CUsmSZptf4Ldnjwn+4uKkihXD1y8AAAAgCgSc1TMuLk4ul0uWZcnNlDiEmj3il98HDPY0z0qVTBZQJzCSDQAAgBjhV+B39OhRTZkyRZdddpnOOussrVmzRuPHj9eWLVtU0h6RAULhdFM9K1WSnntOGjIk+LbmzjX1/Nq3D/5aAAAAQAHg89DI3XffralTp6pGjRrq27evpkyZovLly4eyb4DX6aZ6VqvmTNAnSSVKmIye8fHOXA8AAACIMJ8Dv5dfflkpKSmqU6eOFi1apEWLFuV53IcffuhY54ATTjfV00l2Efdt28y6QQJAAAAARDmfA79evXrJ5XKFsi9A/u6+W+raVTrrrLxf/+kn6dgxqW5dKTk5uLaqVDHB3vHjJlNosFlCAQAAgAhzWdRiiCoHDhxQcnKy0tPTlZSUFOnuFBzXXCN9+qn0yitS//7BXy8lxUz3XLbM1PUDAAAAChh/YoOAs3oCBYpTxdtt9nTPrVuduR4AAAAQQQR+iA4bNkhvvCF9/nner9uBn1PTMlNSzJbADwAAADGAwA/R4ZtvpH/9Sxo//uTX3G6zFk9yLvA791zpwgslMtcCAAAgBjhU6RoIsVNl9dy9W/J4pLg4qWJFZ9p76CHzBQAAAMQARvwQHU5VwN2e5lmpEqUXAAAAgDwQ+CE6nKqA+44dZhuKsgtut/PXBAAAAMKMqZ6IDqea6tmggfTvf0ulSzvX3oEDUuPG0s6dps2iRZ27NgAAABBmBH6IDqea6lm3rvTAA862V6qU9Pffpoj7tm3SmWc6e30AAAAgjJjqieiQPfCzrNC353J5a/lt2RL69gAA8NfIkVJaWt6vpaWZ1xFdgvmdcj/gNAj8EB0qVJCmT5c+/vjk15Yvl374Ie9poMGgiDsAoCCLj5eGDz/5zX5amtlPwrPoE8zvlPsBp8FUT0SHokWl66/P+7W77pJWrpQ++0y66irn2qSIOwCgIBs2zGyHD/c+t9/kjxrlfR3RI5jfKfcDToPAD9HPLufgdFZPpnoCAAq67G/2n3xSysriTX60y/47HTnS1Cq+/Xbv/u+/l158Mf/z+/ThfkCeCPwQPT7+WNq3T+rSRSpb1uw7ftwUcJekKlWcbY8RPwBANBg2zPsmPzGRN/mx4MorTfDm8Zjnl17qfW3rVul//8v/3Ndek957z3s/PPxwaPuKqEHgh+gxYID0119mPZ8d+O3ebf4oxsebdYBOql9fuvBCqUkTZ68LAIg9I0ea/4vyCrrS0kxd2FAl10hLM2/y4+LMNi0tdMFfJL/PwmL9eunii81jl8sktVu2TOrZ0+w75xzpP//J//zVq71BX1aWdPbZZl+JEqHuOQo4Aj9Ej7yKuNvTPCtXdn7R8oUXSl9/7ew1AQCxyU6sIeUMirKvsQoF+/qS+SC0Xbu8++GUSH2fhcXWrVKrVtKhQ2YJy7p10rhx5mdboYL5mderJw0enPf5aWnShAnm93DNNVKLFtKGDVLDhtKPPzpb8xhRh8AP0SOvIu6hWt8HAIA/sq/LsixvdsVQJtbIHvTZihQx7YUq+Mv9fT70kPTccyQQccI//0jNmkkHDkjlyplRuqSkvJO25CWv+23hQqlDB5OvoGFDadUq82E5CiUCP0SPvIq424Gf0+v7sjt+3GwT+OcCADiFYcPMNL0RI8ybb7c7tMGQ2y21by/NmydVqiTt2mWmBH7+uff1UMgeiIwYYR4T9AWvVCmpWjUzPXPlypxLWOyf7al+p3ndbxdcIH33nXT++dLOndJFF0lz5ki1aoXkW0DB5rKscFTDhlMOHDig5ORkpaenKykpKdLdCa8rr5RmzZLeeku67Tazb80a8wesZk2pe3fn2+zYUZo715SK6NTJ+esDAGKH253zQ8LEROno0dC22aKFtGKFNHGiNGSI+XD0p5/MOrBQstfXS2Ydmp2EBMHxeEyA5vRMpo0bpcsuk/74w1x7zhypUSNn20BE+BMbUMAd0SOvqZ7nnGPmuYci6JPMf9oeD5k9AQCn9/773sd2Yo3cxbSd9OefJuiLi5OuvVY67zyzf+nS0LVpu/de72PLCu33GcvcbpOF0x7Ji4sLzfKVunWlJUukxo3NlNJ//nG+DRR4BH6IHnlN9Qw1avkBAHzh8XiDoXbtzEifvdYuVEHRzJlme9FFUsWKZlqfJH3zTWjas9kJRGz164f2+4xVliXdfbfUv7/Uq1fo26tWTVq8WPryS++9gkKFwA/R4447pKlTpa5dvfsWLjTlHQ4fDk2bduDHiB+AgmDkyPzfXKelkUY/knr2lPbskYoWlT780Ox7+GHzOwlVUGS3c+21Znv++Wa7c6fzbdnsBCKNG5vnI0ZIv/4a+iA3Fu/9xx+XXn3VO2IbDmXLektFSCbT50cfhadtRByBH6JHmzZSjx7mk0XbjTdKLVuaxfShQBF3AAWJnUo/9xtg+82402Vt4BuPR1qwwDx+5BEpOdkERFWqSE2behO9OOn4cZMMJDFR6tbN7LvkElPf9quvnG0rOzuByAsvmKUWXbqY/cOGheb7tMXavf/CC9LTT5vHL78cuiUrp7J1q8ll0L27yZ+AmEeaQkSv48fNf3BS6LJ6MtUTQEGSPZviP/+Y7UsvkUo/0v75Rzr3XOnbb6X77jP7MjOlffuk996Tpk93vs2EBJO9MyPDW+e2WDHzFUrZR9Yuu8z7+M8/pUcfDV0AlldJg1CXywhVsfp33vHW4Rs9Wrr99mB6GbgqVaSrrpLefFPq29fcp7NmnXxcMN8rChRG/BA9Nm+WZszwFlXftcvMj4+Pz5ny2En2iN+2bWQsA1AwDBsm3XOPGTEoW5agryAoV06aPdtMeSxTxuy7+Waz/eQTKT09dG3bQV+kWJbUurUpD7BiRWjbGjZMeuIJc88XLRr6ez8Uo4yffmqCLMkEfw8/HHw/A5WQIL3+uvTAA+b5l1+aUePsCf+jdUQVeSLwQ/T46ivp+uul5583z7PX8IsL0a1crZp04YVmGsSRI6FpAwD8Zb8xsyzz5q2gBn2xuC7rVLLPPmna1BTMPnrUuxbPKRkZ+S9B+OknU7D78sudbTO76dNNqaNDh0wph+rVzf7Zs0PXpm39evN/flaWmeYaynvfnr5qB5hut++jjPnd+3Fx5meWmmoK37tcIeu+T1wu0w972unixSaQ93hCP6KKsCPwQ/TIndXTDvxCkfbYVqSIGWF87z2pRInQtQMAvjp0yHxKbzt+vOBmU4y1dVm5WZb073/nnUzF5fKO+r33nrPtzpxpZqTceuvJr5UsaQq6L1wYmg8sLUsaNMhM8/zuO7OvY0ezDXXgl5lpgk6Px/z/nJVlgpJQGjbMfI0YYT5kGT7c1L87dkyaMsXMCMpLfvf+ypXm32y3bqH70NpfLpc0dKipBSlJ339vgmqCvphTQO44wAeRCPwAoKC5+WYzipSUZN7433VXwU2ln33ExO5fLI0ifPaZ9OCDUpMmeRdqv+kms50/X/rrL+fa/eADs61T5+TXateWKlUygckPPzjXpm3TJmnHDhMY2HUD7cDv22+l/fudb9P2r3+ZoKl0abOWMi7OBGShvvdzf0Cxdq1p86abTJBtW73aTN989VWpbVuT6Gf4cBMob9yY894fMSK0fQ7EnXdK775rvl+3O/Qjqgg7krsgeuQu4L5jh9mGKrFLdsePm//U7T4AQCSkpXlTrw8dat5YSubvYPakFwXJsGGmzIH9hvf48dgI+izLO9rUp49Zc5Zb7dqmxMI335hyRPZaqmBkZHhH1vIqAeBymRptH35o2r3wwuDbzG7RIrNt1UoqXtw8rlnTZNxev94EQqHIUJmWZn6GktSvnwlOPB6zxj+U977HY5KfSGbE7/hx6YorzPTWX3+VzjnHe+zSpWbtbXYlSkjjxpkvqeDf+5s3e4O+rCzzcy/I/YVfGPFD9Mg94nf11dJ//hP62jejRnkXkQNAJLndJuAbO9YEGzY76UWoUukH6rvvzN/o8ePN8+PHzRv2xx+PbL+cMGuWGVErUUIaMiT/4+65xwToV1/tTLtffmmmcNapY0Ya82LX81u61Jk2s7MDv0suybn/iivMNlTTPbOyvMH1tdeapCilS5sPFbp1C929/9RTJrP3jTd6p5Z++aWZavv111KzZt5jmzY1o3tXXGGCYclMzbYV9BG07COSR4+GvjYjws9CVElPT7ckWenp6ZHuSvitXm1ZkmVVrhzedseONe1ef3142wUAX2zaZFl9+1rWPfdEuic5zZpl/nbm9dW8uWW53aFpd8QIyxo1Ku/XRo0yrwfL47Gs1q3N9zJkSPDX80fPnqbdBx/M/5hly8wx5cubvjopJcVc+6uvcu7/4guzPyXF+TYty3s/Va7svXeeecbb5uHDzrc5apS5fu77Kb/9uWVkWNZdd5ljExN9OydSsn9PHo/5fd5yi2U98kjB7jf8ig0Y8UP0yD3VM1zsWn4UcQcQSdlTrGe3fbuZivbqq3knGQmX48elDRu8z9u3l+rVM/XtJDN68MYbZiriihVmJCUUwpFQ5quvzGhm8eKnHu1z2tGjZl2hdOrZLueea0bH9u7N+TsJ1h9/mNGvhATvqKLtkkuk/v1N5u387tVgzJxpttmTogwcaLJvb9niTUziFLdbWrbMm9wlO1+L1T//vOlXNIygud3eaagul/TYY9L//mdGNn35XhEdwhCIwkGFesTv4EHLeu01y5o61Tz//HPLWr7cso4eDU179qfGy5ebT7uqVvW+5tSnxgDgqzfftKzzzrOsmTNz7vd4LKtNG/N36tFHnW/3dCNojz5qWePHW1bt2mbkJSsr57m5Rwt++MGynngitKMIuUdkfB2h8UX2n/f99/t2zrFjlvXZZ2ZUNpjRsM8/9/5/dLoR06uusqzLL7esVasCby+3d94x7Z93nnPX9NVrr1nWBRdY1ty5J++XLKtcOcvav9+59iZONNdt2jSw31mwo4WR9u9/m35ecEGke4LT8Cc2IPCLMoU68MsuK8uyXC7zR2n37tC0Yf9xfughs3W5TLvR8kcbQGyxpxaOGXPyax9+aF4rXdp8SOak/P7mDR1q9pco4Z3CWb68Zf30k/cYX4LGQYMsa98+Z/tsWZbVp09optgdOWJZAwean/X27b6dk5lpWSVLmn4sWRJ42+npljV5smW98Ubg1wiG222WXQTzPTjt2DHLatDAskqVsqwFC5y55r59JpCULOvFFwO7RjimHIfStm3e91mbN0e6NzgFAr8YRuD3/7ZuNX+MEhJCt07EsrxveOLjzXbwYII+AOFnr3FOSLCsnTtPfv34ccuqV88c88ILzrefPfjbvt2yzj8/55q9WrUs66WXTIDjj1tvNeefc47vQdTp7NljWb17m+vGxXmDv+wjkU7IyPDv+F69TF/uusvZfhQkHo9ZX/jEE6FZc5efVauc/RB44EDzu2rc2ASWhVW7dubn8NRTke4JToE1fohd8+aZFNXr15vnVaqEtgBq7nn8zz9f8FMxA4g9r75qtt26mRptucXHe0sFvPCCqeHmpOz1+GrWNGUCJJPK/r33zDqye+4xGS798eCDUuXK0po10kUXmVTygbIss9axfn3p7bfNPo/Hm5b+zDOl554L/Pq5+Vve55ZbzHb6dNOfcNm1Szp8OHztXXedqVH39dfOXM+ypMmTpb//zv+Ypk1NWQcn/Pyz9N//msfjxpn1jIWVfc++915o1m0i7Aj8EF169DD1gb791jwPR/H2YcO8wWWRIgR9AMIrM9MkWZBM8oz89OolVaxoEl28/77z/Rg2zARRx46ZQPOLL6QffzRFrAN9c3zOOdKSJabe3aZNpubcL7/4f51160zB7H79pH37vMGxnVSja1eToOuhh0zSikDfxD72mLR8eWDntmtngty//w6s5MHzz0tPP21+v766/HLT5vz5/reX26efmnvs00/zP8blMm1KzpV1WLlSuvlmE7j78oHGnDmBf4BgWaYcg9ttPmRp3z6w68SKa681SYLWrjX/1hH1CPwQXUqWNNvffjPbcAR+aWneT42PHSuY2bgAxK5p06QDB0zdtnbt8j+ueHETnI0c6X3z7ZQ9e0zQk5Vl/ha63aaGncsV/LXPPNMEf40bmwylF1/sX3D12WdSaqq0eLEZcbz8cjPKlX12xsyZ0mWXmcdPP21GJz0e//q5aJE596KLzM/DX/HxUs+e5vF77/l3rsdjAr/HHjOjo76qVs1s7RHaYHz6qfTuu946fvnp2NFsnQr8PvzQbC+7zHz4eiqPP25+/489FlhbH31kZhYVLWrqBBd2yclS585Sw4anHnEtLEaOzP89YFqaeb2gC8PUUzio0K/xa9TIzDe315fcfXdo2wtlZjgA8MWpkrqEg8djWXXrmj7072/2heJv4d69ltWqlbcum68Zm/fvN7XdOne2rD/+OHVSjc6dvesSb77Zv3V/9nqnO+/0/ZzcfvjBXKNYMZOoxVd2Xb5SpfxbO2dnvLzkEr+7epKzzjLX+uSTUx+3d683Kci2bcG326CBudbkyac/duVK7+935Ur/2/r9d8vq1s2yHnvM/3NjVUZGaOoyRqMCmqmV5C4xrNAHfi1bmn9cFSqY7ZNPhq6t3P+Q3W7LWrHCsh54gOAPQHh4PJY1Y4ZlXXFF3kldwuHKK72JZX7+2bs/FG92Dh60rK5dLevbb/MP4HbssKwOHSxr+HDvvr/+8v3N6eTJ5nuRTCDoSyD19dfm+CJFLOvPP31rJy8ej2XVr29ZZ59tWWvW+H7egw+a9m+80b/2fvnFnFe8eHDJbbZv92a3/uef0x9vB/DBZh9du9abnMfX9z12gfuOHQNvN5RJ4xDd7L97dimXAjAgQHIXxC57Mf1115lpL/bUnVDIXsxUknr3lpo3l8qUoZgpgPBwucy65lmz8k7qkp8vvjAFtVevDq79NWu8U/bGjjXTMW2+FrH2R8mSZlpm69beQuyPPGJe83ikl1+WatWS5s7NuRawalXfp5327Gmm9BUrZoqwb99++nNGjTLbPn1MQetAuVxm2uWaNdLZZ/t2jmV5pzueqmh7Xho0MP9nHT4c3L2weLHZpqZKpUuf/vgrrjDbYKd72kXb27eXkpJ8Oyctzaw5nT1bWrDAt3Ny38OhTBoXrQ4dkr7/PtK9iLx+/cy/qRdeMFOChw+PrqR/YQhE4aBCP+J31VXmk5XXXw9/2//9r2m7bdvwtw0A/ujRwzudMVCHD5uRKcmM+kViutcdd5j2L77YFA23p/FVrWpmYARj4ULfipt/8413xDMS9cx+/NE7PTSQGo32/5tjxwbeh7vuMte47z7fjl+61Bxfp05w903z5uY6r73m33kDBpjzWrXyrf2bbjLlNpwqKRJrfv3V1KFMTg5vmY5Q8Ke+YkaGqZGa/f7zeCyrenXv36LExFD21ieM+CF22SN+mZnhb9vO7vXNN+aTLwAIpQ8+kJ580rcRqdwefNBsp071Lwtkdo88YlLbV6xoyiQ4kcjFX5Urm+3ixd5szldeab6nZs2Cu/Yll5gyALaFC6W//jr5uCeeMNvevc1oo1MyM72liU7FHu3r2NGb4Mwf559vtkuX+n+uzU7ocsklvh3fqpVpb/36wO+bXbukVavM6Ns11/h37rBh5v3C8uXen19+liwx5SLefVfauTOwvsa6evXMKFd6uplNEM3smQS5k7SkpZn9hw9Lr79uktqUL29G2R95RDp+3BzncnnvR7tUTBQl/SPwQ3Tp10+aMMFkuFu+3P+sbMGoV0+qUcP8I1+yJHztAiicnn3WvIG1Szn4o3lzkwHU7TZTNP311VemhpkkvfWWf9NMnTRihAl+bYmJ0uefmzdvTlq2TLrqKpOxc9Mm737LMm/yateWHn3UufbmzDE/05tvPv2xWVkm4PN3mqft8sulO+80ZTcCceSIyaYZF2d+Pr5ISDABZzA18CpVMh96fPCB+fDB33MHD5bq1j11vUW3W7r3XvP4X/+Szj038P7Gsri4wDPSFjTZa5LaAZsd9NWoYWp93n67yRZ85Ij5t3/rrd4P/NPSTJ1Hu1RM7msVdGEYgYSDCv1UT8syC+vtRfbhXoDdp49p+8EHw9sugMJl1Srv37lduwK7xqxZ5holS/qWkCO7vXtNdsOBAwNr20l28oTExNAlUdi82bLOPNP788qdeMXtPnkaWDB277as+HjT3rp1pz/+8OHIT7Hbvz+y7fvr0KHTJ7R59VXzO0hONr8T5M+ecpyY6P/fk4Io99+Viy/2Tt9s3ty8/tNPOacKx0BWT0b8EH3saU9VqoR/AXaHDmY7d2542wVQuLz6qtl26+b/aIetY0dTID0jwyRF8Ue5cmakJdK1zOxP4kP96XqtWmYmR8WK5ufVsqVJ/GJ76inTrlMjjRUqeOvd+TKCUqyY+Yqk5GT/jj92zIw01q0r/fNPaPp0KsWLn7ru3z//eEdxR440vxPkr0kTk5AoK8v8bYh2Dz7onaqZmChNnCiNH2+mkf/wgxkZPOecnFOVcyf9s4Ui0VWohCEQhYMK/Yjf1q3eRebnnRf+9nfu9H4itGdP+NsHEPsyMky9Nsmy5s4N7lpvv22uU7myZR05cvrjf/ut4NTsisSn63//nTNxwz33WNbIkaFpb/Lk0ydA+f13Z9o6csQkqZkzx/9zfa2nmJeGDc33OH26f+e98YZlXXqpKWUSrKNHLeullyzr5Zdz7r/vPtO3hg2DK3VRmIwebX5ml14a6Z4Ez67LaZd2ieISXYz4IXbNmGE+lZFM+u5wq1TJpPBdsMD/Tz8BwBfTpkkHD5qRkksvDe5aN95okqH85z+nH61at86k6+/Zs2AksIrEp+tly5qfQ5065vn48WY0KBTp2q+5xqw/+/13b+Ka7LZuNf1o3NisNQrGp5+aNXd20h9f7dtnyjdccklgfbBHNf0t6/D+++b/2d9+87/N3GbOlAYONAk67JHHQ4fM+wnJrIE91cggvOx1fosWSbt3R7YvwRgyRJo/3zx+993oW6cXhCBW3QIRkD2jWSQCP0kaNCgy7QIoHF55xWxvvz346ex2MpTTOXrUJP84fFjavz/y0wolE3DlJ5Q1s0qWlNauNUGZ221+hqFo74wzzFTe//3PfLVpk/P1jz4y27Jlg/992Jk9f/rJfKhQqpRv5339tbkndu0KrA9XXGECq9mzzRiqLxk+9++X5s0zjwNNaJPdL7+YKby7d0vPPCONGSOVKGH233abydR9+eXBt1MY1KwpvfGG+SAg0CnokZaW5p3CfsEFUo8e3vty+HCzjZaafAFgxA/RJXt2ripVItcPAAiFo0fNCE/p0uZNabg8/rgp8F2+vMniWdgLWD/7rDfoC2W69ltuMdv33z95BDPQou15qVrVrGP0ePIeXcyPv2Uccrv4YhMwbttmRlJ98fnnZn1go0ZS/fqBtZtdkSLe0alx47wlO1580QTXTmeIjXV9+0pnnhnpXgTujz/M1uUy94Ad9EXTOr0gFPK/7Ig62QO/SI34SdKsWdJ990l//hm5PgCIPUWLmpp5O3Y4+4l6RoaZpp5XSv+5c6V//9s8fuMNPlQLV0IZydSHffZZk0gmewCyZ4+pXSiZUUEn2KN+33zj+zl24Ne2bWBtFi9ugj9J+vJL385xMuCVzBt6uxbjkSOmdqP9Ow3FFF4UXB6P9OOP5nG/fifXAh027NQzDWIAgR+iw8iR5j/d7IGfPS0mLS38/1BHjzafFH31VXjbBVA4OD3VMj1devhhacqUnNkq//7bFCaXTAZGfwtlx5rsQZ8dEORV98spCQlm3V3t2jn3f/yxeZParJlzReMvuMBsfS3knp5uRoGlwEf8JP/W+R06ZD5YlZwL/CTzu/vXv8zjvXtNfUiCvsAtWSJ16eINqKPF229LK1ZISUk564MWIgR+iA7x8eYP9/Tp5nmtWmYKiP2fdLinarRvb7aUdQDglKVLpZUrQ3PtatW8xcKfe867/847TYmc+vUjX7qhICgo6dqdHvWSvCN+337r2/exZIkJPuvWDW6GzRVXmGs0aXL6Y2fPNmsKa9UyI3NOeu017xTmhASCvmD89Zf0ySfSpElm7Wa0sD/IGD7cJOsrhAj8EB3s/3Tt2laZmXl/Mhsudj2/efPMf4wAEKzBg6Xmzc1Uz1AoXtxsP/xQ2rjRPL7rLjPadOmlZsphYTdyZP7/n4RyGtiXX0qdO0tTp5qRNvtDxe7dnWvjnHNMUpeDB6Wffz798cGu77M1aiRt2JDzA4f8lCkjdepkstH6kgjGH2lp5v/rIkWk48cLRQbHkOnc2SRB+uMP/6YO52bP5spLKGZzjRtn+jtwoLPXjSIEfogew4ZJQ4eax//8E9n5+a1amT96f//tnS8OAIFavVpavty8Kb366tC0Ya/dsyzv6F67dmaq58svk+Qikr77TvrsMzOCUry4KUHwyCNSgwbOtREfb0a9lizx7brNmpk3+Fdc4VwfTqdtW+mLL8xyCidl/6A4K6tQpe8PiRIlvKPR770X+HXs2Vy5fw+hnM3Vpo1J2lRYhaGuIBxU6Au4W5ZlJSaaYpuJiZHtx9VXm348+2xk+wEg+t11l/l7csMNoW2nTx9vcfJdu0JbDB2+GTHCW0w8Ls6ydu70vjZqlHk9Fhw9alnLloW/3fzuce794MyebX5+5cpZVlZW4Nexfw8PPWRZv/8emt/Ls89a1h9/OHe9AoYC7ohdaWnm07pQp9j2Bev8ADghM9PUcZOkO+4IbVtvvGHqwklm3RaZDSMvPt5MQate3UxFnDrV7I/UGvZQOHzYrKlq00basiXvY+bPN0XrnVZQ1m3GmnbtzO/07799S9yTn2HDpBtuMFPN69Rx/m/Sl19KDz1k1pimpztzzShG4IfoEc4U276w1/lt2cI6P8AW7jUb0Sj3z2jqVLPuqm5dMw0vlD8jl8tkjHS5QlucHL6zA5Bt28zzQYOkAQNCF5RPm2auv3Nn/sesWuVsuaLixaWGDc3jvIIEt9uUGklJCW7NWF4itW4z1iUkmLWYUuDTPY8fN1lt7cR9NqdKyhw7Jt1/v3n8r39JycnOXDeKEfghOoQ7xbYvGjeWfvtNWruWYseALRJrNqJN7p+RnbSqTh2TZj7UP6MFC8xkz4IwcwLGsGGm3Ibtv/8N3Ujs00+b65+qrMM995jMmlOmONfuqco6LFsm7dollS4ttWjhXJsIrVtukVq3NsmhArFli1lfbLPfS91+uzRhQvD9mzBB+vVXqUIF8zcXBH6IEgVxqobLJdWr53zmMSCa5fWBTCQz8BZE2X9GQ4eakZ64OFMXNNQ/o4I2cwJeY8Z43/gWKRK6++B09fwOHZK+/948bt3auXbtwG/uXDPSk51dvqJz58KdeCPatGhhyoP07x/Y+XXqmN+5ZP4WHT8uXXiheR5s4rw9e7yjuU8/zWifLQxrDuEgkrsUUB5PpHtgEgDktxg6lhIEIDo88YQ3CRMJFPJmJzEI18+IJBcFW7juh//9z1y/deu8X58717xevbqz/7cdP25ZZcqYay9d6t3v8VhWzZpm/8yZzrWHgun11y1rwQLzOK+/PR6PZd1/f/D/Bvr3N9c491xz78UwkrsA4XL0qKmzVKmStH9/ZPvCFDsUJImJZvTCTsb0+OPSv/9tPsWNpoK/oZKVJd17r3e6ZTjW2hXEmRMwwjkSaxdyX7nSJF3JLXv9PidntMTHS5ddZh5nn+5prycsUUK6/HLn2kP4/POP9PrrZrpufo4eNcmr/vUvk8xl1668/ya5XNLzz3v/Jq1cKd13n39/n1avNqVLJJM4ifc/XmEIROEgRvwKoPr1C84nlfanZ489Zlk//MAn+YiMbdssq0gRc+8lJJitXa5AsqxatSxr0CDLWrTI+0lsYRqx3rbNstq0say6dRkVRfhHYj0ey6pSxVx70aKTX7/4YvPaq686265lWdYbb5hrt2rl3ff442bftdc63x7C48ILze9w3Li8X9+yxfzOJctyuSzrqacsy+0+/XUzMiyrcmVz3vXXm5Igvjh40LwPuu0237+HKOZPbEDgF2UI/AqgAQPMH6UBAyLdE8N+s2B/8WYS4Zaaau69lBTzJtO+Jxs0sKxixXLen+XLm9pyd99dOKYhLlpkWRUrer//gQPN/lj7PuG7SHzo0b27ud9Gj865//Bhyypa1Ly2fr3z7W7fbllPPmk+mLQ1bWra+9//nG8P4TFu3MkBvW3+fMuqUMG8XqaMZX35pX/XnjHD+0Fip06WlZnp+7kFYRlOGBD4xTACvwJo5kzzB6l+/Uj3xFi61Pum0uXy7VM1nF5hGpEKxu23e++/77/37rcDm8cfN/9meve2rLJlvcfOmeM95sEHLevvv30PhqLhd+PxWNYLL1hWfLz3ex40KOcxBH8Il+efN/da37459y9caPZXqhS+N83p6ZY1ZYpl7d8fnvbgrBEjTPF1+2/bb7+Z/R6PZV1xhXkfIpkA//ffA2vjyy8tq3hxc52LLzb3TF6OHi2U73lY4weEU9u2Zi3T+vXeOkyRNGKE97FlST17Rq4vsYQ1lKfn8Uiff24e9+mTMy27vY4sPl7q2lWaNMms8Zg/Xxo82Kwnso957jmpXDnzcz3rLOnAAVN4/JtvzFqS3Ar67yYz09Qou/9+s07lnHPM9/rCCzmPY60dwqVXL+mvv8y/q+xSU6X335dGjw5fxuqkJFMPjqyL0Sk+3hRfr13bPJ882WzT0kzxdMuSbr3VZJG1j/FXx44m63FSkrR4sdS+vSkcn9vo0SYT7fLlgbVTGIQhEIWDGPEroFq3Np9ETZoU2X7YIwb33GNZ//qXd2ThiSci269YkXtEhhGanN5+2/w8Spa0rB07Ar9OXFzO6aC5vw4c8B67YIH5NHjw4IL7u+nc2bve8cUXC830IyBPhw+bEb577+XfQqzIvsSkXj1vVufHH7esd95x7ve8YoVllStnrn333Tlf+/NP76jgtGnOtBclmOoZwwj8CqjHHjN/bG65JXJ9yP1Gd98+yypVirV+Tgt3Cv5o4fFYVpMm5mcyZkzg18n98+3c2bxBvPxyy6pRw6SYz659e+89bq8DsQPHYcOC+56csnKlZdWubVlffx3pngCRNWKEZQ0f7n2DXqyYWWdorwUuCNOyERg7SU+o33P88otJ9DJ0aM42evTwTgV94olCdS8x1RMIt8svNymymzePXB/27jVTyey0yGXKmGlj//2vme7G9DFnDBsW3hT80cLlMmngR46UBg0K7Bp5pbT/9FOpfHmT/n3LFmnjxpznnHmm1LChlJAgHTtm9nk8ZvvKKycXig4Ht1v64Qfv83PPNVPB7cLEQEEwf77UqZOZai1JP/9s/g1+913o2oyPN/+uq1Uzz48ckb74QnryyYIxLRuBS0szy14k8/c4VP83NmokTZ8uFS/u/f9ixgxp2jTz/1DjxmbJC/dS3sIQiMJBjPghXzfcYBZXT5wY6Z7EtiFDco5IFZRRpWgXbEr7rCwzMph9xK9hw5zH9OtnWePHW9bWrd59gSaGye+8v/82ZRri4izru+9O3Wcgkj75xJtt17Is6+mnw1NWIXfm6U6dmD0RC+zfq13CJxy/z9z3UosWhfJeYsQPKGx+/dUsyHe7pQsuyPsYt9v8aUTg0tJMEXLJjPjZ+0JRZDmaLF0a/L0VbHHxMWOkF1/0HvvEE9K6dd7fzW+/mUQW99wj1aghtWwpPf20SRAQSGKYvBLKrFplRiA3bjSffBeEZE9Aftq0MdtffzX/DrIXbg+lYcOkgQO9z2fNyvvfPqJH9tkax46ZbV5/V5324INSlSre5z/8wL10OmEIROEgRvwKuH/+saxvvgl/u7feaj7l6to179ffe88suJ471/m2oyGVvhPsTxbtheVnnmm2NWsWyk8YT5g3z3z/7dt7i7GHmy+jhbt3W9Zzz1nWBRd404vbX/bv1D5/2DDz/JFHLGvv3pO/7CLC9vWHDrWst97yftJdpoxl/fhjWH8EQEDq1zf37MyZJimTFL57107/n5gYnvYQGsHO1gjW/v2F/l5ixA+IhPXrTQr6yy/3rjUKh02bvOmTH38872OWLZM2bDDrKJxW0FPpO8Xtlu66y3wyXqyY9MknZlTnzz+le+8tnGsojx/3rudr2DByv2tfRgsrVJCGDJGWLJG2b5defVW68kqzTvPvv03Zk+HDpaJFvffymDFmfWHurwULvNfv0sWkEO/Tx/w8zjrL/Jts0iS8PwMgEPYMkfHjpYwMqWxZ6eyzQ99uWpr5d2mvly7ssyaiWbCzNYL14ovcS/4IQyAKBzHiV4C53d6RgyVLwteuXbahU6f8j9myxZvxMBSZBQtLmQN7fd/115vnV11lng8ZEtl+Rcp//2u+/7Jlzdq2aJSebllTp1pWZqZ33Wb2Iut5fX35pff8117z7o+PL5TFgxHF3ngj573dpUvo2yws/18g9LiXLMuinENMI/Ar4G64wfzRGTkyPO39+ac3oFu69NTH9u9vjrv88tD0ZeDA8C/sDie327KqVfNOi7Isy/roI/O8QgXv9L/C4u+/vR90jB8f6d4EL3cZiSeeML/zvL6y16Sy61VR3gPRZsQI799t++v5581roZqmH+lpgYgd3EsnMNUTiJQOHcx23rzwtPfjj2ba4aWXmnISp/LII2Yq3ldfScuXO9uP9HSTbl8y091isczB119Lf/0lJSebFOiSdNVVZmH5nj3Sxx9Htn/h9sQTZopk48bSHXdEujfByauMxIgR0lNPmem8ub9cLu95I0bkPC8cCQ0AJ8THSy+9ZP6m2S65JLTT9CM9LRCxg3spMGEIROEgRvwKuE2bvKNeBw+Gp819+yxrwwbfjr3tNtO/q692rn2320wPyj0dLtY+bbOnefbrl3P/Y4+Z/ffeG5l+RcIvv3inQ86ZE+neBCfQT435tBmxwL5fR460rOXLzZb7F4gq/sQGCZEOPAu7bt26aeHChWrfvr1mzJgR6e4gWHXqSLVrS5s3S4sXm+QRoVamjPnyxdCh0jvvSJ99ZlLdN2wYfPtjxnhHu4oXlw4flnr1Mp8YS7Ez8vfMM9I115gEPtkNGCDdcEPhSuaRmSnVry/Vq+cd5Y5Wp/rU2H7dyfOAgsS+X4cPN+VNsrJIhw/EMJdlUdgrkhYuXKiDBw/q7bff9inwO3DggJKTk5Wenq6kpKQw9BB+699feu016f77peefD00be/ZIP/0ktWvnnXbmqzFjpNatpbZt/T83Lx07mumjXbpIt95qpg21aWO+d3v6HG8iYs/x42aKb+5AGED0KVrUBH2JiWbaMoCo4U9swIhfhLVt21YLFy6MdDfgpNtuk1q0MGUdQuWFF0wK+TvukF5+2b9zH3nE2b60aWNS2L/0Us79sTTyYVm+Bcn//COVKiUlFII/rQkJBH1ALEhL8wZ9djp8PqwDYlJUJneZOHGimjRpoqSkJCUlJalNmzaaNWuWo20sXrxYnTt3VtWqVeVyufTRRx/ledyECRNUq1YtFStWTK1bt9Zyp5NmIPqcf74Z9atVKzTX37fP1FySgp9Kevhw4OdlZprHI0eeHPTZhg0zr0ezjAwzffeee6RDh/I/bvBgqWpV6fPPw9e3cPvvf6Vnn2VEAIgVeSU2IkERELOiMvCrXr26xowZoxUrVuiHH35Qu3bt1KVLF/3yyy95Hr906VIdy6Og9tq1a7Vr1648z8nMzFRqaqomTJiQbz+mTZumwYMHa8SIEVq5cqVSU1PVsWNH7d69+8QxTZs21dlnn33S1/bt2/38roH/9+KL0sGDZk1Z586BXcPjkR591AQqv/3m37mWZUYazztP2rjx5Ne//NIUyl67NrC+FTQff2yKtM+ebdYw5ichQTpyxEzzjUW7dpnR4ocflj78MNK9ARCs7EGfPcJnZ0Qk+ANiU8hTzYRJmTJlrNdff/2k/W6320pNTbWuu+466/jx4yf2//rrr1alSpWsZ5555rTXlmTNtOt2ZdOqVStrwIABOdqqWrWqNXr0aL/6vmDBAqt79+4+HUtWzyixfbupbfbSS85eNz3dskqXNlnXpk8P7lqdO5vr9O7t33njx3uLVc+ff/LrnTqZ18eODa5/BcWVV5rvZ/jwUx+3fr05Li7OsrZuDU/fwqlfP/P9tWhBkXIgFowYceqstaGo4wfAcYWqjp/b7dbUqVOVmZmpNm3anPR6XFycvvjiC61atUq9evWSx+PRpk2b1K5dO3Xt2lUPPfRQQO1mZWVpxYoV6pAto11cXJw6dOigZcuWBfz95GfChAlq1KiRWrZs6fi1EQJr1pipgc8+a0bInDJhgrR/v8nG2b17cNeyP+H93/9MFlJfLF0qDRpkHj/7rKkfmNuFF3qPjXZ79njrE/bseepjzzrL1MDyeKQ33wx930Jp5Micn/avXOn9npo2NSMCAKLbyJH5r+WLhWn6AE4StYHfmjVrVLJkSRUtWlR33nmnZs6cqUaNGuV5bNWqVTV//nwtWbJEN910k9q1a6cOHTpo4sSJAbe/d+9eud1uVapUKcf+SpUqaefOnT5fp0OHDrr++uv1xRdfqHr16vkGjQMGDNDatWv1/fffB9xnhNGFF5qF8lu3Shs2OHPNzExvltBHHzWFpIPRsqXJyOl2m0yfp7Njh3TddSabY48eJmtpXi64wGyXLHE26I2EGTPMz6dZM6lBg9Mff/vtZvv669Gd1CY+3jvVy7Kk++4z23POMd9bKAo7AwCAkIrawK9+/fpavXq1vvvuO911113q3bu31p5iTVFKSoreffddTZs2TQkJCXrjjTfkciKVfZDmzp2rPXv26NChQ9q2bVueo5aIQiVKmCQvkjR3rjPX/OMPk0XxzDOlG2905pqPP262b71lgtT8ZGVJ118v7dwpNW5s3vzn9++nZUupSBETKP7xhzP9jJTJk832ppt8O757d1NTcetWU+IiWmVf59OjhwniixQxI9mU5wAAICpFbeCXmJiounXrqnnz5ho9erRSU1M1bty4fI/ftWuX+vfvr86dO+vQoUO6P7/RCh+VL19e8fHxJyWH2bVrlypXrhzUtREj7GnATgV+jRtLv/xirudUuYALLzT1/I4dM1M38/PEE2bqZlKSNHOmVLJk/seWKGFGyKTonu65ZYsJeFwuE/z4olgxU8tQMsFxNBs2zPze33/fPD92jKAPAIAoFrWBX24ej0dH80kxvnfvXrVv314NGzbUhx9+qHnz5mnatGkaMmRIwO0lJiaqefPmmjdvXo4+zJs3j1E7GHbgt2CBc9P+4uOdLxNhv5F/+21TuiAvd99tRjD/9z+pXr3TX9Ne57dkiTN9jISEBOnBB83avurVfT/vzjulp5/2ltyIZo8/bkb6JDN1maAPAICoFZVVhocOHapOnTopJSVFBw8e1OTJk7Vw4ULNtpMwZOPxeNSpUyfVrFnzxDTPRo0aac6cOWrXrp2qVauW5+hfRkaGNmZLVb9582atXr1aZcuWVUpKiiRp8ODB6t27t1q0aKFWrVpp7NixyszMVJ8+fUL3zSM6jBxpRoqSk00ylpUrzRRIyaybcrt9XziflSW98450yy1mRMlpl14qPfOMdMMN+Y/kVasmff217+sKL7xQ+s9/pPXrnetnuFWteupR0Pw0bGi+YsFTT5mRPgo7AwAQ/cKQZdRxffv2tWrWrGklJiZaFSpUsNq3b2999dVX+R7/1VdfWYcPHz5p/8qVK62t+aRdX7BggSXppK/euVLfv/TSS1ZKSoqVmJhotWrVyvr222+D+t5Oh3IOUWLUKJP6vkEDy0pIsKxJk3Luzy+Fdl5ee82c07y5ZXk8oelvXv7+27I++iiwcw8etKyNG8PbXzgr970ayL0LAABCyp/YwGVZ0Z52r3A5cOCAkpOTlZ6erqSkpEh3B6diF8d97DHpySe9WRJPt05q5EgzpXPYMJNBs3596fffTUbPjAz/Rgt9lb3Nv/+WSpeWrr7aFGPv2NEUay9Mqb0//NCsVezQIfD1lDNnSi+9ZH6meZW9KMjse/e880wG2SuvNPdHXgWfAQBAxPgTG0TlVE8gKthvjIcPl557zkyVk6SJE826vwYNzFf9+mZbo4aZSmmn0pekmjVN0Fe+vLRvnwkgQ1FDzW5z8mRp+3bptttM0JeQYOrY2SUaCgPLMmv7fv9dmjrV98QuuX31lfk9V6oUfYGf223ugUmTpP79zT0hee/paC5VAQBAIcWIX5RhxC8KFS1qgr74+FO/YZ4+3ZRMkEzdtBdfNOUb/v7bjDzNnRvakZZRo6QRI/LeH0ibv/xigkmXy9TDixbffWdGus44Q9q1y2wDsXKl1Ly5WR/3118meI8mjzxi1n726mUS/wAAgALHn9ggZrJ6AgVSWpoJ+hITTdD36KMmsHj7bfP42mulRo1M5sTsBcJr1DDbv/8221AHfZIJ0m64Iee+YNpMSDBTJj//XMon426BZNfu69o18KBPMiUtmjUzv/9333Wka2Fl1yG8/PLI9gMAADiCwA8IlezroY4eNdunnzZTJ3v1MhkTP/jAjIwdOmTq9NnKl/cWgJfCl0p/yhRv5s5g2zzrLPN9HDliRr+cNHKk+fnmJS0t8PWIx49L06aZx74WbT+V228329deM1NIAxGq7/VUdu+WVq0yj+2yJAAAIKoR+AGhkFcSjGHDzHM7yUt2CQk5SyXcdpt0xRXmcfZU+qH21FOSx+NMmy6Xd22g04Xc7TWJuftn/9zj4wO77oIFZnpnuXLSZZcF38+bbjJJYtatk775JrBrhOp7PZW5c802NdWsUQQAAFGPwA8IBbc772mSdvB3uuQYeY0W5vXm30mhaDNUhdzzCqKdyDhpT/O84QZv4fJgJCV5k8O89lpg1wjV93oqTPMEACDmkNUTCIVTTb873Rv1/EYLJW+2T6ff7IeqTXvE75tvzFRHlyv4vtqy9++JJ/IPtn1lWWZkTnJmmqetf39pwwbvCG4gsn+vTz5pRmNDueZzwwazJfADACBmkNUzypDVsxDIXlMvt7S00Nfxc7LNo0dNTcAjR6T16826Pyf98YdUu7Z57HJJe/dKZcsGfj3Lkn76STrnnJxTbyPp99/N9+hyeTPEJiaGNmGOZUkbN0opKaZNAABQIPkTGxD4RRkCP0SdDh2kzExp3DipVStnr3377dLrr3ufly4tLVokNWnibDuRsGqVSQb0wQfSnDlmnWT2chsUUQcAoNCjnAOAgmPOHGnZMueDvrQ0E/Q98YS0eLEJ+vbvl1q0MIXX/XHkiHT4sLP9y23PHuk//5FWrz71cUuXSldeaUpBzJhhRt+eesoEfXbm1wsuCN2aTz4LBAAgJhH4AQgtJ9f12bKvSRw+XLroImnTJunMM6Vjx6SePU3ZDF9NnWqyV9rrGZ02cqQZ+RwyRPrvf3O+lpZmgrqvvpIuucQkxJk1y0w1vekmacAAk2101ChvoPfnnybgdTr4O3rU1JDs3l1KT3fuugAAIOII/ACEx8GDZmTNCW63dMcd0gMPePeVLWvWEV54oVSvnn/lGCZPNv0L1Xq2+HizdlAytRIzMszj7CUZ7rvPjFwWKWKmsP72m/Tee6YWoj2ts1Mnkyl02zbp0kt9yxDrj6VLpb/+Msl4mEoOAEBMIasngNC78Ubp/felDz+UunQJ/npDh0qVK5uA7dtvpUaNzP74eOnrr00hdjs5y5EjJlHJ2Wfnfa2dO6V588zjnj2D71tehg0zUyhHjDBB3//+Z9qcMcMb1J11lvTddyaYrV7de272pDrFiknduklvv20CyNyjh8GyyzhcdlloRmoBAEDEMOIHIPRKlzaF4Z2q5/fZZ2Y9X1KSVL/+ya8n/P9nWpYl3XmnWV9o1+jLbfp007fzzpPq1HGmf3kZPtxbHuGuu0zQd/XV3gQtN94ovfBCzqAvL3Zw+v77Zlqrk+bMMVvKOAAAEHMI/ACEnl3Pz6nA7513zPaWW8woX36OHpV27TKJW26+WRo82IwGZmcHhE7W7svPu+/mfH7NNf5fo317qUIFU7rCHql0wp490sqV5nGHDs5dFwAAFAgEfgBC78ILzXbFiuCzZ+7ZI33xhXncq9epjy1WzIwOPvqoef7CC1LdutIjj5jnmzaZ6ZVxcdINN5g1d07XSMzulVfM1h6R3LnT/2skJJjpoe+84w2onTB3rtk2aWKm0QIAgJhC4Acg9GrVkqpUMVMTv/8+uGtNnWpG7Vq08K7tO5X4eFMO4YMPpJIlTUbMZ54x0y2nTDHHtG8vvfqqN9FKKGTPRHrsmDcjaSBZOe+8U7r1VqlUKef6Z6/vY5onAAAxicAPQOi5XN5Rv6VLg7uWPc3zdKN9uV17rRndq1fPPH/5ZWn3bhN4VajgDcpCURQ9e9BnX3/YsOCCP6ede67Upo10xRWR7gkAAAgBsnoCCI8LLjAJSYJZ5/fHH9IPP5jpjjfe6P/5jRpJy5dLt91mEs689JKUmChlZYUu6JNMyYW8rm8/D6Qkw+7d0qRJ0j//SKNHB91F3Xuv+QIAADHJZVmWFelOwHcHDhxQcnKy0tPTlUSdLUSTNWtMgHL55SbwCtTvv5sSDk4kYyla1AR9iYkmEUw0WbVKatbMfA+7d1N3DwCAQsif2ICpngDC45xzTAbNYII+yZRccCLoS0vzBn1ZWQVjuqU/mjY1pSyOHpU+/ji4ay1daspjAACAmEXgByA6ODk5Ifuau6NHC9ZaO1+5XN6afnaSmkAcPWpGYcuVkzZscKZvAACgwCHwAxA+Ho+0dq20YIH/595+u9SliykJEYxoSLTiK3ud45w5pq5fIL75Rjp0yCS4qVvXub4BAIAChcAPQPjMmyc1biz16+ffeRkZZlTrk0/MtMxgnCrRyqhRgSVaiZT69U02zuPHpRkzAruGXcbhssvMKCIAAIhJZPUEED6tW5ti6Zs3S9u3S1Wr+nbehx+aUam6daXzzguuD6cq0B6qrJ6h1LOnSfQyZYqp7+cv6vcBAFAoMOIHIHySkqQmTcxjf+r5Za/dx6hUTj16SCVKmCD6+HH/zt2zxwSNktShg/N9AwAABQaBH4Dwsgu5+1rPb+tWaf588/iWW0LTp2iWkmLW902ZYuob+mPePJM0p0kTqUqV0PQPAAAUCAR+AMLrggvM1tcRv/feM8HJxRdLtWuHrl/RrHjxwM5jmicAAIUGa/wAhJcd+K1ebZK2lCyZ/7GW5Z3m2bt3yLsW9daulcqWlSpX9u34ESPMmsmWLUPbLwAAEHGM+AEIrxo1zPREt1v67rtTH+t2SwMGSBddJF13XXj6F63uvttkTH31Vd/PqVlT6t/fZAYFAAAxjcAPQPg9+6z0xRdSq1anPi4hwQR+ixebxDDIX+vWZjtlirPF7gEAQExwWRbvEKLJgQMHlJycrPT0dCXxRhiA7cABqWJF6ehRk6mzadNTH5+WJpUpY7KCVqgQli4CAABn+RMbMOIHoGBassRMW9y/P9I9iQ5JSdJVV5nHU6ac+tijR6UxY6SBA009RQAAEPMI/ABExty50tCh0q+/5v36Cy9Id9whPflkePsVzXr2NNupUyWPJ//jli2TDh2SKlWSzjknPH0DAAARReAHIDL+8x8z6jRnzsmv7dsnffqpedyrV3j7Fc2uukoqVUrassUEd/mxyzh06CDF8d8AAACFAf/jA4iMU9XzmzZNOnZMSk01xcXhm+LFpa5dzeMZM/I/jvp9AAAUOtTxAxAZF15otkuWmCyULpf3Nbt2H6N9/hs0SOreXbriirxf37tXWrnSPL7ssrB1CwAARBaBH4DIaNXKlGv46y8zNbFmTbP/t9+kb781UxBvuimyfYxGzZqZr/zMm2cC7XPOkapUCV+/AABARDHVE0BklCjhDVCWLPHuf/dds+3YUapcOfz9inV//GECbqZ5AgBQqBD4AYicvNb57dplRvt6945Mn2LBoUPS449LzZub0g3ZPfywSZ7z8MOR6RsAAIgIAj8AkWMHfj//7N336qvStm1Sly6R6VMsKFZMmjTJrOWbNevk10uVomg7AACFDIEfgMi5/HJp3Tpp0aKc+6tUMcELAhMXJ/XoYR5nL+budkemPwAAIOII/ABETqlSUoMGJqPn4cPS1q2R7lHssIu5f/qplJFhHnfrJrVsKS1eHLl+AQCAiCDwAxAZI0dKaWne5x99ZDJ79u1r9o8cGaGOxYjmzaW6dU1A/fHHUlaWNH++9MMPUlJSpHsHAADCjMAPQGTEx0vDh0sDB0o33mhKN1iWtHmz2R8fH+keRjeXyzvqN2WKtGyZlJkpVawoNWkS2b4BAICwo44fgMgYNsxshw/PuX/hQmnUKO/rCFzPnmb0dPZsqUYNs++yy8waQAAAUKgQ+AGInGHDzCjfiBHefQR9zhg50oyaXnqpdOaZZiqtZAK/tDST6IXptAAAFBp87AsgsoYP945AJSQQ9DnFnkp76aXSmDGmPqJkSmcwlRYAgEKHET8AkZWWJnk8UpEi0rFj5jnBX/CyT6X9+WczslqxovTvfzOqCgBAIcSIH4DISUszgcmoUSbr5KhR5nn2bJ8I3LBh5mc6fboZVd29m6APAIBCymVZlhXpTsB3Bw4cUHJystLT05VESnZEs+xBX/ZAJL/9CFzRoiawTkyUjh6NdG8AAIBD/IkNmOoJIDLc7ryDO/u52x3+PsWitDRv0JeVxVRaAAAKKQI/AJFxqoySBCbOyD16aj+X+BkDAFDIEPgBQCzKa8ps7tqJBH8AABQaBH4AEIuYSgsAALIhuUuUIbkLAAAAAMm/2IByDgAAAAAQ4wj8AAAAACDGEfgBAAAAQIwj8AMAAACAGEfgBwAAAAAxjsAPAAAAAGIcgR8AAAAAxDgCPwAAAACIcQR+AAAAABDjCPwAAAAAIMYR+AEAAABAjCPwAwAAAIAYR+AHAAAAADGOwA8AAAAAYlxCpDsA/1iWJUk6cOBAhHsCAAAAIJLsmMCOEU6FwC/KHDx4UJJUo0aNCPcEAAAAQEFw8OBBJScnn/IYl+VLeIgCw+PxaPv27SpVqpRcLpcj1zxw4IBq1KihrVu3KikpyZFronDiXoITuI/gFO4lOIV7CU5x+l6yLEsHDx5U1apVFRd36lV8jPhFmbi4OFWvXj0k105KSuKPGRzBvQQncB/BKdxLcAr3Epzi5L10upE+G8ldAAAAACDGEfgBAAAAQIwj8IOKFi2qESNGqGjRopHuCqIc9xKcwH0Ep3AvwSncS3BKJO8lkrsAAAAAQIxjxA8AAAAAYhyBHwAAAADEOAI/AAAAAIhxBH4AAAAAEOMI/Aq5CRMmqFatWipWrJhat26t5cuXR7pLKOAWL16szp07q2rVqnK5XProo49yvG5ZloYPH64qVaqoePHi6tChgzZs2BCZzqJAGz16tFq2bKlSpUqpYsWK6tq1q9avX5/jmCNHjmjAgAEqV66cSpYsqe7du2vXrl0R6jEKqokTJ6pJkyYnCiK3adNGs2bNOvE69xECMWbMGLlcLg0aNOjEPu4l+GLkyJFyuVw5vho0aHDi9UjdRwR+hdi0adM0ePBgjRgxQitXrlRqaqo6duyo3bt3R7prKMAyMzOVmpqqCRMm5Pn6s88+qxdffFEvv/yyvvvuO51xxhnq2LGjjhw5EuaeoqBbtGiRBgwYoG+//VZz5szRsWPHdPnllyszM/PEMffff78+/fRTvf/++1q0aJG2b9+ua6+9NoK9RkFUvXp1jRkzRitWrNAPP/ygdu3aqUuXLvrll18kcR/Bf99//71eeeUVNWnSJMd+7iX4qnHjxtqxY8eJryVLlpx4LWL3kYVCq1WrVtaAAQNOPHe73VbVqlWt0aNHR7BXiCaSrJkzZ5547vF4rMqVK1vPPffciX379++3ihYtak2ZMiUCPUQ02b17tyXJWrRokWVZ5t4pUqSI9f777584Zt26dZYka9myZZHqJqJEmTJlrNdff537CH47ePCgVa9ePWvOnDnWJZdcYt13332WZfE3Cb4bMWKElZqamudrkbyPGPErpLKysrRixQp16NDhxL64uDh16NBBy5Yti2DPEM02b96snTt35rivkpOT1bp1a+4rnFZ6erokqWzZspKkFStW6NixYznupwYNGiglJYX7Cflyu92aOnWqMjMz1aZNG+4j+G3AgAG66qqrctwzEn+T4J8NGzaoatWqqlOnjm6++WZt2bJFUmTvo4SQXh0F1t69e+V2u1WpUqUc+ytVqqRff/01Qr1CtNu5c6ck5Xlf2a8BefF4PBo0aJAuuOACnX322ZLM/ZSYmKjSpUvnOJb7CXlZs2aN2rRpoyNHjqhkyZKaOXOmGjVqpNWrV3MfwWdTp07VypUr9f3335/0Gn+T4KvWrVtr0qRJql+/vnbs2KEnnnhCF110kX7++eeI3kcEfgCAiBswYIB+/vnnHGsgAH/Ur19fq1evVnp6umbMmKHevXtr0aJFke4WosjWrVt13333ac6cOSpWrFiku4Mo1qlTpxOPmzRpotatW6tmzZqaPn26ihcvHrF+MdWzkCpfvrzi4+NPyiC0a9cuVa5cOUK9QrSz7x3uK/jjnnvu0WeffaYFCxaoevXqJ/ZXrlxZWVlZ2r9/f47juZ+Ql8TERNWtW1fNmzfX6NGjlZqaqnHjxnEfwWcrVqzQ7t271axZMyUkJCghIUGLFi3Siy++qISEBFWqVIl7CQEpXbq0zjrrLG3cuDGif5MI/AqpxMRENW/eXPPmzTuxz+PxaN68eWrTpk0Ee4ZoVrt2bVWuXDnHfXXgwAF999133Fc4iWVZuueeezRz5kzNnz9ftWvXzvF68+bNVaRIkRz30/r167VlyxbuJ5yWx+PR0aNHuY/gs/bt22vNmjVavXr1ia8WLVro5ptvPvGYewmByMjI0KZNm1SlSpWI/k1iqmchNnjwYPXu3VstWrRQq1atNHbsWGVmZqpPnz6R7hoKsIyMDG3cuPHE882bN2v16tUqW7asUlJSNGjQID355JOqV6+eateurWHDhqlq1arq2rVr5DqNAmnAgAGaPHmyPv74Y5UqVerE2obk5GQVL15cycnJ6tevnwYPHqyyZcsqKSlJAwcOVJs2bXTeeedFuPcoSIYOHapOnTopJSVFBw8e1OTJk7Vw4ULNnj2b+wg+K1Wq1Ik1xrYzzjhD5cqVO7Gfewm+GDJkiDp37qyaNWtq+/btGjFihOLj49WzZ8/I/k0Kac5QFHgvvfSSlZKSYiUmJlqtWrWyvv3220h3CQXcggULLEknffXu3duyLFPSYdiwYValSpWsokWLWu3bt7fWr18f2U6jQMrrPpJkvfXWWyeOOXz4sHX33XdbZcqUsUqUKGF169bN2rFjR+Q6jQKpb9++Vs2aNa3ExESrQoUKVvv27a2vvvrqxOvcRwhU9nIOlsW9BN/06NHDqlKlipWYmGhVq1bN6tGjh7Vx48YTr0fqPnJZlmWFNrQEAAAAAEQSa/wAAAAAIMYR+AEAAABAjCPwAwAAAIAYR+AHAAAAADGOwA8AAAAAYhyBHwAAAADEOAI/AAAAAIhxBH4AAESRtm3batCgQZHuBgAgyhD4AQAAAECMI/ADAAAAgBhH4AcAQBT7/PPPlZycrPfeey/SXQEAFGAJke4AAAAIzOTJk3XnnXdq8uTJuvrqqyPdHQBAAcaIHwAAUWjChAm6++679emnnxL0AQBOixE/AACizIwZM7R7924tXbpULVu2jHR3AABRgBE/AACizLnnnqsKFSrozTfflGVZke4OACAKEPgBABBlzjzzTC1YsEAff/yxBg4cGOnuAACiAFM9AQCIQmeddZYWLFigtm3bKiEhQWPHjo10lwAABRiBHwAAUap+/fqaP3++2rZtq/j4eP3nP/+JdJcAAAWUy2JxAAAAAADENNb4AQAAAECMI/ADAAAAgBhH4AcAAAAAMY7ADwAAAABiHIEfAAAAAMQ4Aj8AAAAAiHEEfgAAAAAQ4wj8AAAAACDGEfgBAAAAQIwj8AMAAACAGEfgBwAAAAAxjsAPAAAAAGLc/wFwdNL8BTy9cAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.figure(figsize=(10,6))\n", "plt.plot(num_ks, valid_misclassification, 'rx--', label='Validation')\n", "plt.yscale('log')\n", "plt.xlabel('k')\n", "plt.ylabel('Misclasification rate')\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "seihHrN44uia" }, "source": [ "You can now go back to your implementation of leave-one-out and train your kNN model using the best k that you found. How does it perform compared to the VEGA kNN? Do you got a better model? Why?" ] }, { "cell_type": "markdown", "metadata": { "id": "IrYX_LjKkjcE" }, "source": [ "## Challenge - kNN QSPR for predicting BCF 🥇\n", "\n", "\n", "Develop a kNN model for regression to predict the bioconcentration factor (BCF).\n", "\n" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "id": "rv1bQTxtVvuz" }, "outputs": [], "source": [ "if 'google.colab' in str(get_ipython()):\n", " df_bcf = pd.read_csv(\"https://raw.githubusercontent.com/edgarsmdn/MLCE_book/main/references/BCF_training.csv\")\n", "else:\n", " df_bcf = pd.read_csv(\"references/BCF_training.csv\")\n" ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "id": "X5a-AgMZ55Rj" }, "outputs": [], "source": [ "# Your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## References\n", "\n", "```{bibliography}\n", ":filter: docname in docnames\n", "```" ] } ], "metadata": { "colab": { "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.15" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "0d5dd3b8b7914bafb3c57066f2117f4b": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_bbb549238aca44c0bcc7b41a4e0516d4", "placeholder": "​", "style": "IPY_MODEL_34b5cde70d7a41ae82eec617f48a33ff", "value": "100%" } }, "131bcbae8f1a4166b7c187e27752a4a1": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_0d5dd3b8b7914bafb3c57066f2117f4b", "IPY_MODEL_da3943a1533d48339cc99add56f296ec", "IPY_MODEL_8ab9c22f3c6b407aab7c2b20542caddf" ], "layout": "IPY_MODEL_62cc78dd0f354e4bab79b1adcbbb8d51" } }, "164fc49ebc3b466f9b4c25260637210a": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "24b693f1c368469faf63263a9c520158": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "34b5cde70d7a41ae82eec617f48a33ff": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "3f07d346263c40ae80952f23251b4b74": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_d19e4dface4c4701a08ac745a31b8225", "placeholder": "​", "style": "IPY_MODEL_24b693f1c368469faf63263a9c520158", "value": " 49/49 [01:48<00:00, 2.54s/it]" } }, "4bfe0903581b4ca999017a19607380d5": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "536147b20e874c8a820f2ffbfd010b5a": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "62cc78dd0f354e4bab79b1adcbbb8d51": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "7976f19eb8f1414e8de608ed280755a7": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "8ab9c22f3c6b407aab7c2b20542caddf": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_536147b20e874c8a820f2ffbfd010b5a", "placeholder": "​", "style": "IPY_MODEL_cbeab16ad6e344eea77e78270864e9bc", "value": " 50/50 [00:16<00:00, 3.10it/s]" } }, "914eff8c2fae4f248af2dd1c0ab2adec": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_c90eb39b1d6e4b5ba8136df7e5480a61", "IPY_MODEL_b5249ba10245474e8ea18b502bebffb2", "IPY_MODEL_3f07d346263c40ae80952f23251b4b74" ], "layout": "IPY_MODEL_7976f19eb8f1414e8de608ed280755a7" } }, "99446b25af0a41efb78e8347a0468bc5": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "b01c55a604c741d3838d98d412f6098e": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "b5249ba10245474e8ea18b502bebffb2": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_4bfe0903581b4ca999017a19607380d5", "max": 49, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_ef2b2ce93832431aac24fdb8dcbaf7f2", "value": 49 } }, "bbb549238aca44c0bcc7b41a4e0516d4": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "c90eb39b1d6e4b5ba8136df7e5480a61": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_d23639c2608d4c4bbbb58fbeeff6c934", "placeholder": "​", "style": "IPY_MODEL_164fc49ebc3b466f9b4c25260637210a", "value": "100%" } }, "cbeab16ad6e344eea77e78270864e9bc": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "d19e4dface4c4701a08ac745a31b8225": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "d23639c2608d4c4bbbb58fbeeff6c934": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "da3943a1533d48339cc99add56f296ec": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_b01c55a604c741d3838d98d412f6098e", "max": 50, "min": 0, "orientation": "horizontal", "style": "IPY_MODEL_99446b25af0a41efb78e8347a0468bc5", "value": 50 } }, "ef2b2ce93832431aac24fdb8dcbaf7f2": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } } } } }, "nbformat": 4, "nbformat_minor": 1 }