{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第10章 图像分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在前面的章节中，我们学习了一系列图像处理算法。这些图像处理算法能够帮助我们提取图像中的我们所需的信息，为后续的计算机视觉任务打下基础。在这一章中，我们将进入计算机视觉的第二部分——图像内容理解。图像内容理解又被称为视觉识别，其目标是希望计算机对图像的识别结果与人类对图像的识别结果一致。所以视觉识别通常需要一个（机器）学习的过程。自从2021年深度学习崛起以后，深度学习基本统治了视觉识别的所有任务。所以在视觉识别这一部分，我们将主要介绍基于深度学习的视觉识别算法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10.1 简介"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic4.zhimg.com/80/v2-9576cd5719a3a1f7168c2d8ff3c01bfb_1440w.webp\n",
    "\" width=600>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图10-1 图像分类示例。</div>\n",
    "</center>\n",
    "\n",
    "\n",
    "图像分类是视觉识别中的一个最基本的问题，对这个问题的研究也是研究其他视觉识别问题的基础。图像分类，顾名思义，就是根据图像中的物体或者场景，给图像打个标签。如图10-1所示，图像分类就是分别给这些图像打上“猫”、“狗”、“沙滩”的标签。用数学语言来描述图像分类的问题即是：定义图像空间$\\mathbb{I}$和预定义的类别集合$\\mathcal{C}$，给定数据集$\\mathcal{D} = \\{(\\mathbf{I}^{(n)},y^{(n)})\\}_{n=1}^N$，其中$\\mathbf{I}^{(n)}\\in \\mathbb{I}$是数据集中的第$n$张图像，$y^{(n)}\\in\\mathcal{C}$是其对应的标签；图像分类的任务是从$\\mathcal{D}$中学习得到一个从图像空间到类别集合的映射$f:\\mathbb{I} \\rightarrow \\mathcal{C}$，从而给定任意一张测试图像$\\mathbf{I}$，我们可以用学习得到的映射函数$f$预测其标签：$\\hat{y}=f(\\mathbf{I})$。通常这个学习过程需要建模条件概率$p(y|\\mathbf{I})$，则$\\hat{y}=f(\\mathbf{I})=\\arg\\max_{y\\in\\mathcal{C}}p(y|\\mathbf{I})$。一个通用的图像分类流程如图 10-2 所示。首先，我们从用于训练的图像中提取图像表征，并结合对应的标签信息训练一个图像分类的模型（分类器）。对于每一张测试图像，我们采用同样的表征提取过程得到其表征，并利用训练得到的分类器预测其所属类别。\n",
    "\n",
    "\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic1.zhimg.com/80/v2-a126bac75f410c394b7d1e063ed6c060_1440w.webp\n",
    "\" width=800>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图10-2 图像分类流程。</div>\n",
    "</center>\n",
    "\n",
    "\n",
    "\n",
    "由于机器学习中不同类型分类器的性能大同小异，由上图可知，图像表征的提取是图像分类中最关键的一步。图像表征的好坏，直接决定了图像分类性能的高低。传统图像表征提取算法主要基于手工设计的图像特征向量，如我们前面学习的SIFT特征就是一种最典型的手工设计的图像特征向量。传统图像表征中的集大成者是视觉词袋模型（Bag-of-Features）[1-4]，由Andrew Zisserman等人受文本检索领域的词袋模型（Bag-of-Words）启发而提出。该表征模型首先从图像中检测并提取出多个特征点，之后利用聚类算法对这些特征点的特征向量进行聚类得到“词典”，在此基础上通过构建好的词典对每张图像中的特征向量进行编码，从而构建每张图像的表征，即视觉词袋模型表征。最后，基于视觉词袋模型表征，训练图像分类器。可以看到，传统的图像表征提取与图像分类器的训练是分开进行的，两个部分的信息彼此之间无法进行交互影响，因此分类性能会受一定的限制。随着深度学习的崛起，通过深度卷积神经网络（CNN）联合图像表征学习和分类器学习逐渐成为了主流，因为通过联合学习，能够大幅度提升图像分类的性能。第一个深度卷积神经网络由Yan Lecun于1989年提出 [5]，用于手写字符的识别。随后在1998年，他改进了这个网络。改进后的网络被称为LeNet-5 [6]，已具有所有现代深度网络的元素。2012年，Geffory Hinton带领他的学生构建了 AlexNet [7]模型，是一个8层的卷积神经网络，在ImageNet图像分类比赛中取得冠军，从此开启了图像分类的深度学习时代。2015年，微软亚洲研究院的研究员何凯明等人提出了ResNet [8]，是一种可以堆叠上千层的卷积神经网络，再度刷新了ImageNet图像分类比赛的最好成绩，从此深度学习开始统治整个视觉识别领域。\n",
    "\n",
    "接下来，我们将分别介绍基于传统的视觉词袋模型的图像分类算法和基于深度卷积网络ResNet的图像分类算法。在介绍算法之前，我们需要先了解一些常用于图像分类的数据集以及评测指标（metric）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10.2 数据集与评测指标"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在图像分类任务中，我们通常会选择以下几种常用的数据集进行模型的训练与测试：\n",
    "1. MNIST [9]：MNIST是一个手写数字的数据集，每一个样本都是一张黑白图像，其对应的标签为数字0-9中的一个。它包含了60000张训练图像和10000张测试图像。\n",
    "2. Caltech-101 [10]：Caltech-101于2004年由李飞飞等人提出。该数据集包含连同背景在内共102个类别9144张图像用于训练和测试。\n",
    "3. CIFAR-10 [11]:  CIFAR-10数据集是由 Geffory Hinton 的学生Alex Krizhevsky和Ilya Sutskever整理的一个用于识别常见类别的小型数据集，它一共包括50000张训练图像和10000张测试图像。\n",
    "4. ImageNet [12]： ImageNet是一个大规模图像识别数据库，由李飞飞团队从2007年开始，耗费大量人力，通过各种方式收集制作而成。ImageNet数据集一共包含了14197122张图像，共1000个类别。\n",
    "\n",
    "这里，我们使用Caltech-101作为我们后续实验的数据集。我们随机选择其中的60%作为训练样本，剩下的20%作为验证样本，20%作为测试样本。\n",
    "\n",
    "在图像分类中，我们通常选择准确率作为评测指标。对于$n$张测试图像样本，若模型预测类别准确的样本数为$m$，则该模型的准确率为$\\frac{m}{n}\\times 100\\%$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10.3 词袋模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "词袋模型（Bag of Features，BoF）是一种提取图像表征的方法，它可大致分为以下四步：\n",
    "1. 从训练图像集中提取特征，形成特征库；\n",
    "2. 对特征库中的特征进行聚类，构建“视觉词典”；\n",
    "3. 用所学到的视觉词典对每张图像的特征进行编码；\n",
    "4. 基于特征编码对图像进行表征。\n",
    "\n",
    "\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic4.zhimg.com/80/v2-3276fc1c87f8b59342e3cdf5e8ac130f_1440w.webp\n",
    "\" width=400>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图10-3 对特征聚类形成视觉词典。</div>\n",
    "</center>\n",
    "\n",
    "具体地，首先，从给定的训练图像中提取特征：通常，我们可以提取图像的SIFT特征。根据我们在第7章所学的内容，每个SIFT特征描述符是一个128维的向量。将从训练图像集提取的所有SIFT特征收集到一起，构建一个图像特征库。接着，我们通过聚类，如Kmeans聚类，将这些图像特征聚成$K$个聚类中心。这些聚类中心我们称之为“视觉单词”，所有视觉单词构成的集合即为视觉词典。如图 10-3 所示，图中的每一小块表示一个视觉单词。在此之后，我们需要利用视觉词典对图像特征进行编码。举一个最简单的编码例子：对图像中的每一个特征，我们都可以在词典中找到一个与其最相似的视觉单词来表示这一特征，我们称这个过程为将该特征分配给（assign）这个与之最相似的视觉单词。一个视觉单词被分配的次数越多，也就可以认为这个视觉单词表示的语义在图像中出现得越多。所以，如图 10-4 所示，我们可以用一个直方图来统计一张图像中的视觉单词出现的频率，并最终使用这一信息来形成图像的表征。\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic4.zhimg.com/80/v2-a2787f1c304f8ee4665314f3ee02946b_1440w.jpg\n",
    "\" width=600>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图10-4 基于视觉词典构建图像表征。</div>\n",
    "</center>\n",
    "\n",
    "最终，我们得到所有训练图像的表征。基于图像的表征，我们再训练一个常见的多分类器如SVM等，便可以实现图像分类。接下来，我们将动手编写实现基于视觉词袋模型的图像分类算法。我们先导入必要的库，并导入Caltech-101数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "import cv2\n",
    "import numpy as np\n",
    "import os\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.kernel_approximation import AdditiveChi2Sampler\n",
    "import random\n",
    "import math\n",
    "from imutils import paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-17T08:19:58.800786Z",
     "iopub.status.busy": "2022-11-17T08:19:58.799615Z",
     "iopub.status.idle": "2022-11-17T08:21:11.642963Z",
     "shell.execute_reply": "2022-11-17T08:21:11.641717Z",
     "shell.execute_reply.started": "2022-11-17T08:19:58.800689Z"
    }
   },
   "outputs": [],
   "source": [
    "# 图像数据\n",
    "data = []\n",
    "# 图像对应的标签\n",
    "labels = []\n",
    "# 储存标签信息的临时变量\n",
    "labels_tep = []\n",
    "# 数据集的地址\n",
    "image_paths = list(paths.list_images('./caltech-101'))\n",
    "\n",
    "for image_path in image_paths:\n",
    "    # 获取图像类别\n",
    "    label = image_path.split(os.path.sep)[-2]\n",
    "    # 读取每个类别的图像\n",
    "    image = cv2.imread(image_path)\n",
    "    # 将图像通道从BGR转换为RGB\n",
    "    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "    # 统一输入图像的尺寸\n",
    "    image = cv2.resize(image, (200,200), interpolation=cv2.INTER_AREA)\n",
    "    data.append(image)\n",
    "    labels_tep.append(label)\n",
    "\n",
    "name2label = {}\n",
    "tep = {}\n",
    "for idx, name in enumerate(labels_tep):\n",
    "    tep[name] = idx\n",
    "for idx, name in enumerate(tep):\n",
    "    name2label[name] = idx\n",
    "for idx, image_path in enumerate(image_paths):\n",
    "    labels.append(name2label[image_path.split(os.path.sep)[-2]])\n",
    "\n",
    "data = np.array(data)\n",
    "labels = np.array(labels)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们接着对数据集进行训练集验证集和测试集的划分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train examples: (5486, 200, 200, 3)\n",
      "x_test examples: (1829, 200, 200, 3)\n",
      "x_val examples: (1829, 200, 200, 3)\n"
     ]
    }
   ],
   "source": [
    "(x_train, X, y_train, Y) = train_test_split(data, labels, test_size=0.4, stratify=labels, random_state=42)\n",
    "(x_val, x_test, y_val, y_test) = train_test_split(X, Y, test_size=0.5, random_state=42)\n",
    "print(f\"x_train examples: {x_train.shape}\\nx_test examples: {x_test.shape}\\nx_val examples: {x_val.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-14T04:15:42.707808Z",
     "iopub.status.busy": "2022-11-14T04:15:42.706668Z",
     "iopub.status.idle": "2022-11-14T04:15:42.717775Z",
     "shell.execute_reply": "2022-11-14T04:15:42.716426Z",
     "shell.execute_reply.started": "2022-11-14T04:15:42.707764Z"
    }
   },
   "source": [
    "我们先进行视觉词典的构建。为了得到视觉词典，我们首先需要对输入图像的表征进行提取，这里我们使用SIFT对图像进行处理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 构建一个词典，储存每一个类别的sift信息\n",
    "vec_dict = {i:{'kp':[], 'des':{}} for i in range(102)}\n",
    "\n",
    "sift = cv2.SIFT_create()\n",
    "for i in range(x_train.shape[0]):\n",
    "    # 对图像正则化\n",
    "    tep = cv2.normalize(x_train[i], None, 0, 255, cv2.NORM_MINMAX).astype('uint8')\n",
    "    # 计算图像的SIFT特征\n",
    "    kp_vector, des_vector = sift.detectAndCompute(tep, None)\n",
    "    # keypoint和descriptor信息储存进词典中\n",
    "    vec_dict[y_train[i]]['kp'] += list(kp_vector)\n",
    "    for k in range(len(kp_vector)):\n",
    "        # des使用kp_vector将其一一对应\n",
    "        vec_dict[y_train[i]]['des'][kp_vector[k]] = des_vector[k]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着，为了保持类别之间的平衡，我们为每个类别选取相同个数的特征用于之后的聚类。我们先统计每个类别的SIFT图像特征个数，找到拥有最少特征个数的类别，然后以该类别的特征个数为标准个数，从每个类别的特征中筛选出标准个数的特征进行后续的聚类任务。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-16T07:30:27.467701Z",
     "iopub.status.busy": "2022-11-16T07:30:27.467387Z",
     "iopub.status.idle": "2022-11-16T07:30:56.448159Z",
     "shell.execute_reply": "2022-11-16T07:30:56.446862Z",
     "shell.execute_reply.started": "2022-11-16T07:30:27.467673Z"
    }
   },
   "outputs": [],
   "source": [
    "# 设置最少特征点的数目\n",
    "bneck_value = float(\"inf\")\n",
    "\n",
    "# 确定bneck_value的数值\n",
    "for i in range(102):\n",
    "    if len(vec_dict[i]['kp']) < bneck_value:\n",
    "        bneck_value = len(vec_dict[i]['kp'])\n",
    "\n",
    "# 按照每一个SIFT特征点的响应值对特征进行降序排序\n",
    "for i in range(102):\n",
    "    kp_list = vec_dict[i]['kp'] = sorted((vec_dict[i]['kp']), key=lambda x: x.response, reverse=True)\n",
    "    \n",
    "    \n",
    "        \n",
    "# 为每个类别选择同样多的特征点用于聚类。特征点个数bneck_value\n",
    "vec_list=[]\n",
    "for i in range(bneck_value):\n",
    "    vec_list.append(vec_dict[0]['des'][vec_dict[0]['kp'][i]])\n",
    "\n",
    "for i in range(1, 102):\n",
    "    for j in range(bneck_value):\n",
    "        vec_list.append(vec_dict[i]['des'][vec_dict[i]['kp'][j]])\n",
    "\n",
    "vec_list = np.float64(vec_list)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，我们使用Kmeans聚类算法将这些特征进行聚类，得到的聚类中心组成视觉词典。这里，我们将聚类中心个数设置为200。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-16T07:30:59.703093Z",
     "iopub.status.busy": "2022-11-16T07:30:59.702575Z",
     "iopub.status.idle": "2022-11-16T07:57:08.852213Z",
     "shell.execute_reply": "2022-11-16T07:57:08.851048Z",
     "shell.execute_reply.started": "2022-11-16T07:30:59.703040Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "N_clusters = 200\n",
    "kmeans = KMeans(n_clusters=N_clusters, random_state=0).fit(vec_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在此基础上，我们利用视觉词典对图像中的特征进行编码，并利用直方图统计编码的结果，最后将直方图归一化后便得到图像的表征。注意到这种基于整张图像特征编码的直方图统计表征会丢失图像中特征的位置信息，即把这些特征的位置任意地改变，其词袋模型表征是不变的。为了克服这个问题，我们引入空间金字塔匹配（Spatial Pyramid Matching，SPM）[1]算法提升词袋模型的表征能力。空间金字塔匹配对图像进行不同层次的划分。如图 10-5 所示，在金字塔的不同层次中，图像被划分成的块数不同，分别为$1\\times1$、$2\\times2$、$4\\times4$。我们对每一层中划分的图像块都通过上述词袋模型构建其直方图表征，最后将所有层次的图像块的直方图连接起来组成一个向量，即是整个图像的表征。\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic2.zhimg.com/80/v2-a9acf5e11663843f15015a1734c427e9_1440w.webp\n",
    "\" width=500>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图10-5 空间金字塔匹配对图像进行划分。</div>\n",
    "</center>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-16T07:57:08.854128Z",
     "iopub.status.busy": "2022-11-16T07:57:08.853790Z",
     "iopub.status.idle": "2022-11-16T08:08:49.010003Z",
     "shell.execute_reply": "2022-11-16T08:08:49.008139Z",
     "shell.execute_reply.started": "2022-11-16T07:57:08.854099Z"
    }
   },
   "outputs": [],
   "source": [
    "def extract_SIFT(img):\n",
    "    sift = cv2.SIFT_create()\n",
    "    descriptors = []\n",
    "    for disft_step_size in DSIFT_STEP_SIZE:\n",
    "        keypoints = [cv2.KeyPoint(x, y, disft_step_size)\n",
    "                for y in range(0, img.shape[0], disft_step_size)\n",
    "                    for x in range(0, img.shape[1], disft_step_size)]\n",
    "\n",
    "        descriptors.append(sift.compute(img, keypoints)[1])\n",
    "    \n",
    "    return np.concatenate(descriptors, axis=0).astype('float64')\n",
    "\n",
    "\n",
    "# 获取图像的SPM特征\n",
    "def getImageFeaturesSPM(L, img, kmeans, k):\n",
    "    W = img.shape[1]\n",
    "    H = img.shape[0]   \n",
    "    h = []\n",
    "    for l in range(L+1):\n",
    "        w_step = math.floor(W/(2**l))\n",
    "        h_step = math.floor(H/(2**l))\n",
    "        x, y = 0, 0\n",
    "        for _ in range(2**l):\n",
    "            x = 0\n",
    "            for __ in range(2**l):\n",
    "                desc = extract_SIFT(img[y:y+h_step, x:x+w_step])                \n",
    "                predict = kmeans.predict(desc)\n",
    "                histo = np.bincount(predict, minlength=k).reshape(1,-1).ravel()\n",
    "                weight = 2**(l-L)\n",
    "                h.append(weight*histo)\n",
    "                x = x + w_step\n",
    "            y = y + h_step\n",
    "            \n",
    "    hist = np.array(h).ravel()\n",
    "    hist /= np.sum(hist)\n",
    "    return hist\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们分别使用SPM提取训练集和测试集图像的特征。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SPM 的一些超参数\n",
    "pyramid_level = 2\n",
    "DSIFT_STEP_SIZE = [4, 8]\n",
    "\n",
    "hist_vector = []\n",
    "for i in range(x_train.shape[0]):\n",
    "    tep = cv2.normalize(x_train[i], None, 0, 255, cv2.NORM_MINMAX).astype('uint8')\n",
    "    # 提取图像的SPM特征\n",
    "    hist_SPM = getImageFeaturesSPM(pyramid_level, tep, kmeans, N_clusters)\n",
    "    # 将提取的特征加入直方图中\n",
    "    hist_vector.append(hist_SPM)\n",
    "hist_vector = np.array(hist_vector)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_test_vector = []\n",
    "for i in range(x_test.shape[0]):\n",
    "    tep = cv2.normalize(x_test[i], None, 0, 255, cv2.NORM_MINMAX).astype('uint8')\n",
    "    hist_SPM = getImageFeaturesSPM(pyramid_level, tep, kmeans, N_clusters)\n",
    "    hist_test_vector.append(hist_SPM)\n",
    "hist_test_vector = np.array(hist_test_vector)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后，我们训练一个SVM分类器，进行图像分类。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-16T08:08:49.012878Z",
     "iopub.status.busy": "2022-11-16T08:08:49.012359Z",
     "iopub.status.idle": "2022-11-16T08:09:55.099782Z",
     "shell.execute_reply": "2022-11-16T08:09:55.098643Z",
     "shell.execute_reply.started": "2022-11-16T08:08:49.012825Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearSVC()"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import svm\n",
    "\n",
    "# transform使用一种特征映射的方法\n",
    "transformer = AdditiveChi2Sampler()\n",
    "transformer = transformer.fit(np.concatenate([hist_vector, hist_test_vector], axis=0))\n",
    "\n",
    "# 构建SVM分类器\n",
    "classifier = svm.LinearSVC()\n",
    "# 将训练的直方图进行特征映射\n",
    "hist_vector = transformer.transform(hist_vector)\n",
    "\n",
    "# 对数据进行拟合\n",
    "classifier.fit(hist_vector, y_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们在测试集上对分类器进行评估，并计算分类准确率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-16T08:09:55.101448Z",
     "iopub.status.busy": "2022-11-16T08:09:55.101110Z",
     "iopub.status.idle": "2022-11-16T08:14:55.571032Z",
     "shell.execute_reply": "2022-11-16T08:14:55.570019Z",
     "shell.execute_reply.started": "2022-11-16T08:09:55.101418Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy 0.6867140513942045\n"
     ]
    }
   ],
   "source": [
    "# 将测试的直方图进行特征映射\n",
    "hist_test_vector = transformer.transform(hist_test_vector)\n",
    "\n",
    "# 计算分类准确率\n",
    "acc = classifier.predict(hist_test_vector)-y_test\n",
    "tep = len(acc[acc==0])\n",
    "print('accuracy', tep/len(y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以发现，利用词袋模型在Caltech-101数据集上图像分类的准确率达到了60%以上。当然，我们也可以通过其他的方式在现有基础上提高模型的准确率，我们可以使用Dense SIFT [13-14] 替换SIFT从图像中提取特征。Dense SIFT算法是对SIFT算法的改进版本，它先对输入图像进行分块处理，再对每一块进行SIFT运算并进行特征提取，感兴趣的同学可以查阅参考文献的论文，这里不做详细说明。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10.4 基于深度学习的图像分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2012年，随着AlexNet的横空出世，基于深度卷积神经网络的方法逐渐成为计算机视觉的主流。对于图像分类这个任务而言，利用深度卷积神经网络可以实现从图像到分类标签的端到端预测，不再需要分阶段的表征提取与分类器训练。\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic4.zhimg.com/80/v2-8d618deef73410777f9b508052801a1b_1440w.webp\n",
    "\" width=400>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图10-6 Resnet残差结构。</div>\n",
    "</center>\n",
    "\n",
    "ResNet网络是在2015年由微软实验室的研究院何凯明等人提出，斩获当年ImageNet竞赛中分类任务第一名，目标检测第一名。在ResNet提出之前，所有的神经网络都是通过卷积层和池化层的叠加组成的（关于神经网络的相关知识，请读者参阅《动手学深度学习》）。人们认为卷积层和池化层的层数越多，获取到的图像表征信息越强，学习效果也就越好。但是在实际的实验中学者们发现，随着卷积层和池化层的叠加，不但没有出现学习效果越来越好的情况，反而出现了梯度消失和梯度爆炸等问题。为了解决上述问题，ResNet提出通过数据的预处理以及在网络中使用BN（Batch Normalization）层，同时，可以人为地让神经网络某些层跳过下一层神经元的连接，隔层相连，弱化每层之间的强联系，如图 10-6 所示，这样的网络叫做残差网络（residual network），这也是Resnet名称的由来。Resnet的结构如图 10-7 所示。\n",
    "\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic2.zhimg.com/80/v2-b6478731107b1ed198b4b3ad6fd13aa1_1440w.jpg\n",
    "\" width=800>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图10-7 Resnet网络结构。</div>\n",
    "</center>\n",
    "\n",
    "我们以使用ResNet18进行分类为例进一步说明ResNet的结构。我们将一张分辨率大小为 $224\\times 224$ 的RGB图像输入进ResNet，首先ResNet通过conv_1层网络将其进行卷积得到一个分辨率大小为 $112\\times 112$，通道数为64的“特征图”。通道数是神经网络中的一个非常重要的概念。我们知道一张图像可以分解成R、G、B三个通道，每一个通道都蕴涵不同的信息。在深度学习中，我们往往通过提升图像的通道数，让其变成一张拥有多个通道的特征图。同RGB通道一样，特征图的每个通道都蕴涵着不同的信息，我们可以通过这种方式极大地丰富一张图像蕴含的信息。那么，我们该如何改变特征图的通道数呢？方法很简单，我们仍然可以通过卷积来完成。对于一个通道数为 $t$ 的特征图，可以使用 $t$ 个大小为 $k\\times k$ 的卷积核同时对特征图进行卷积，这样便可以得到 $t$ 个不同的卷积结果，再将这 $t$ 个结果相加可以得到最终的一个输出。这里可以将卷积核的大小视作为 $t\\times k\\times k$。接着，可以使用 $n$ 个大小为 $t\\times k\\times k$ 的卷积核对特征图进行卷积，便可得到 $n$ 个输出，这 $n$ 个输出便是一个通道数为 $n$ 的特征图（关于多通道特征图卷积的详细内容请参阅《动手学深度学习》）。在此之后，ResNet通过4个模块，每一次将特征图的大小依次进行两倍的压缩，最终conv_5的输出是一个512通道的 $7\\times 7$ 的特征图。在此之后，我们将特征图依次输入进池化和全连接层。全连接层将一个任意通道数的特征图映射到一个维度为类别数目的向量，该向量表示模型预测的类别的概率分布。最终，我们便可以根据这一概率分布，预测图像的类别。\n",
    "\n",
    "为了更深入的了解ResNet，我们先动手编写ResNet34网络。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch as t\n",
    "from torch import nn\n",
    "from torch.nn import functional as F\n",
    "\n",
    "# 残差结构\n",
    "class ResidualBlock(nn.Module):\n",
    "    \n",
    "    # 深度学习中的图像和利用模型提取的特征图往往有很多“通道”（channel），比如RGB图像的通道为3，即R通道、B通道与G通道\n",
    "    def __init__(self, inchannel, outchannel, stride=1, shortcut=None):\n",
    "        super(ResidualBlock, self).__init__()\n",
    "        # 观察图 10-6，我们发现残差结构可大致分为左右两部分，左边是一系列的网络层级，右边是一个跳跃连接\n",
    "        # 定义左边\n",
    "        self.left = nn.Sequential(\n",
    "            # 对应图 10-6 中第一个 weight layer\n",
    "            nn.Conv2d(inchannel, outchannel, 3, stride, 1, bias=False),\n",
    "            nn.BatchNorm2d(outchannel),\n",
    "            \n",
    "            # 对应图 10-6 中的 relu\n",
    "            nn.ReLU(inplace=True),\n",
    "            \n",
    "            # 对应图 10-6 中第二个 weight layer\n",
    "            nn.Conv2d(outchannel, outchannel, 3, 1, 1, bias=False),\n",
    "            nn.BatchNorm2d(outchannel))\n",
    "        \n",
    "        # 定义右边\n",
    "        self.right = shortcut\n",
    "        \n",
    "    # forward() 函数在网络结构中起到举足轻重的作用，它决定着网络如何对数据进行传播\n",
    "    def forward(self, x):\n",
    "        out = self.left(x)\n",
    "        # 构建残差结构\n",
    "        residual = x if self.right is None else self.right(x)\n",
    "        out += residual\n",
    "        \n",
    "        return F.relu(out)\n",
    "\n",
    "\n",
    "# 在这一模块中，我们将实现 ResNet34，其对应的结构可查看图 10-7\n",
    "class ResNet(nn.Module):\n",
    "    def __init__(self, num_classes=102):\n",
    "        super(ResNet, self).__init__()\n",
    "        # 前几层图像转换\n",
    "        self.pre = nn.Sequential(\n",
    "            nn.Conv2d(3, 64, 7, 2, 3, bias=False),\n",
    "            nn.BatchNorm2d(64),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(3, 2, 1),\n",
    "        )\n",
    "\n",
    "        # 重复的 layer 分别有 3，4，6，3 个 residual block\n",
    "        self.layer1 = self._make_layer(64, 128, 3)\n",
    "        self.layer2 = self._make_layer(128, 256, 4, stride=2)\n",
    "        self.layer3 = self._make_layer(256, 512, 6, stride=2)\n",
    "        self.layer4 = self._make_layer(512, 512, 3, stride=2)\n",
    "\n",
    "        # 分类用的全连接层，将一个多通道的特征图映射到一个维度为类别数目的向量\n",
    "        self.fc = nn.Linear(512, num_classes)\n",
    "\n",
    "    def _make_layer(self, inchannel, outchannel, block_num, stride=1):\n",
    "        # 定义shortcut连接\n",
    "        shortcut = nn.Sequential(\n",
    "            nn.Conv2d(inchannel, outchannel, 1, stride, bias=False),\n",
    "            nn.BatchNorm2d(outchannel))\n",
    "\n",
    "        layers = []\n",
    "        \n",
    "        # 给当前网络层级添加残差块\n",
    "        layers.append(ResidualBlock(inchannel, outchannel, stride, shortcut))\n",
    "\n",
    "        for i in range(1, block_num):\n",
    "            layers.append(ResidualBlock(outchannel, outchannel))\n",
    "\n",
    "        return nn.Sequential(*layers)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.pre(x)\n",
    "\n",
    "        x = self.layer1(x)\n",
    "        x = self.layer2(x)\n",
    "        x = self.layer3(x)\n",
    "        x = self.layer4(x)\n",
    "\n",
    "        x = F.avg_pool2d(x, 7)\n",
    "        x = x.view(x.size(0), -1)\n",
    "\n",
    "        return self.fc(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以使用nn.Covn2d()函数实现卷积操作，该函数定义如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nn.Conv2d(in_channels,          # 输入的通道数\n",
    "          out_channels,         # 输出的通道数\n",
    "          kernel_size,          # 卷积核的大小\n",
    "          stride=1,             # 卷积核移动的步长\n",
    "          padding=0,            # 卷积核补零的层数\n",
    "          )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们将动手编写完成基于Resnet34的图像分类任务。首先，我们先导入必要的库。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "import torch\n",
    "import torch.optim\n",
    "import torch.utils.data as Data\n",
    "import torchvision.transforms as transforms\n",
    "import torchvision.datasets as datasets\n",
    "import torchvision.models\n",
    "from torch.utils.data import DataLoader, Dataset\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般而言，在进行深度学习时，为了让深度网络学习的信息更加鲁棒，我们都会对图像进行“数据增强”。简单来说，数据增强是指在数据量比较固定的情况下，通过对原有的数据进行灰度、裁切、旋转、镜像、明度、色调、饱和度变化的一系列过程，用来增加数据量，以提高神经网络的性能。在我们的实验中我们也对图像数据行了增强。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义训练图像增强（变换）的方法\n",
    "train_transform = transforms.Compose(\n",
    "    [transforms.ToPILImage(),\n",
    "     # transforms.Resize((224, 224)),\n",
    "     transforms.ToTensor(),\n",
    "     transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])\n",
    "\n",
    "# 定义训练图像增强（变换）的方法\n",
    "val_transform = transforms.Compose(\n",
    "    [transforms.ToPILImage(),\n",
    "     # transforms.Resize((224, 224)),\n",
    "     transforms.ToTensor(),\n",
    "     transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着，我们把Caltech101数据集封装成类，并生成训练集类、验证集类以及测试集类。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 数据集\n",
    "class ImageDataset(Dataset):\n",
    "    def __init__(self, images, labels=None, transforms=None):\n",
    "        self.X = images\n",
    "        self.y = labels\n",
    "        self.transforms = transforms\n",
    "\n",
    "    def __len__(self):\n",
    "        return (len(self.X))\n",
    "    \n",
    "    # 用于深度学习训练构建的数据类中，__getitem__()函数非常重要，它决定着数据如何传入模型\n",
    "    # 在下面的代码中，我们可以发现，当transform非空时，数据将先经过transform进行数据增强，再返回进行后续操作\n",
    "    def __getitem__(self, i):\n",
    "        data = self.X[i][:]\n",
    "\n",
    "        if self.transforms:\n",
    "            data = self.transforms(data)\n",
    "\n",
    "        if self.y is not None:\n",
    "            return (data, self.y[i])\n",
    "        else:\n",
    "            return data\n",
    "\n",
    "# 生成不同的类用于训练、验证以及测试\n",
    "train_data = ImageDataset(x_train, y_train, train_transform)\n",
    "val_data = ImageDataset(x_val, y_val, val_transform)\n",
    "test_data = ImageDataset(x_test, y_test, val_transform)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将生成的类传入数据加载器中。通常情况下，在训练深度网络时我们一般选择使用数据加载器将数据传入网络中。数据加载器非常像一个迭代器，每次返回batch_size张图像与对应的标签。这里你有可能会问，为什么不一次返回所有的图像呢？batch_size又是什么呢？\n",
    "\n",
    "batch_size表示单次传递给程序用以训练的数据个数。比如我们的训练集有5486张训练图像，设置batch_size=100，那么模型首先会调用数据集中的前100个训练样本，即第1-100个数据来训练模型。当训练完成之后，再使用第101-200的个数据训练，直至数据集中的数据全部输入进模型为止。一般情况下，由于GPU资源以及内存的限制，我们无法将数据集中所有的图像数据输入进模型进行训练，因此我们需要设置一个参数batch_size，每次传入一部分数据进行训练，这样便可以减少内存的使用。同时这样做也可以提高训练的速度，因为每次完成训练后模型都会变得更加精确。\n",
    "\n",
    "在我们的实验中，我们设置batch_size=2048。如果你的计算资源不充分，你可以将其调小一些。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 2048\n",
    "\n",
    "trainloader = DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True)\n",
    "valloader = DataLoader(val_data, batch_size=BATCH_SIZE, shuffle=True)\n",
    "testloader = DataLoader(test_data, batch_size=BATCH_SIZE, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们加载开始加载ResNet34模型。这里我们将直接调用已经封装好的库。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 根据当前设备选择使用cpu或者gpu训练\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "\n",
    "# 加载ImageNet预训练的模型\n",
    "model = torchvision.models.resnet34(pretrained='imagenet')\n",
    "num_ftrs = model.fc.in_features\n",
    "model.fc = nn.Linear(num_ftrs, 102)\n",
    "model.to(device)\n",
    "\n",
    "# 多张GPU并行训练\n",
    "model = nn.DataParallel(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们注意到上述代码中出现了“pretrained”的参数，这是神经网络中的一个重要操作“预训练”。大多数情况下，我们能够用于训练模型的算力和数据都很有限，因此我们希望能够尽量重复利用已经训练好的神经网络以节约训练和数据资源。如果我们在执行预测任务时，能够找到一个曾经执行过相似任务、并被训练得很好的大型架构，那我们就可以使用这个大型架构中位置较浅的那些层来帮助我们构筑自己的网络。借用已经训练好的模型来构筑新架构的技术就叫做预训练（pretrain）。具体而言，我们可以直接借用训练好的模型上的权重来初始化我们的模型。这里，我们使用ImageNet数据集预训练的网络参数初始化ResNet34。\n",
    "\n",
    "我们继续定义训练时需要用到的优化器和损失函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义优化器\n",
    "optimizer = optim.Adam(model.parameters(), lr=1e-4)\n",
    "# 定义损失函数\n",
    "criterion = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "关于优化器的相关知识请读者参阅《动手学深度学习》。我们使用常见的交叉熵损失来计算模型预测的标签与真实标签的误差，其定义如下：\n",
    "\n",
    "$$\\ell = -\\frac{1}{N} \\sum_{n=1}^N \\log p(y^{(n)}|\\mathbf{I}^{(n)}) $$\n",
    "\n",
    "接下来，我们开始编写训练模型、验证模型以及测试模型准确率的函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义训练函数\n",
    "def fit(model, dataloader):\n",
    "\n",
    "    model.train()\n",
    "    \n",
    "    # 初始化loss以及模型准确率\n",
    "    running_loss = 0.0\n",
    "    running_correct = 0\n",
    "    \n",
    "    # 开始迭代\n",
    "    for i, data in enumerate(dataloader):\n",
    "        \n",
    "        # 准备好训练的图像和标签，每次传入的数据量为batch_size\n",
    "        x, y = data[0].to(device), data[1].to(device)\n",
    "        # 我们需要在开始进行反向传播之前将梯度设置为零，因为PyTorch会在随后的反向传递中累积梯度\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        # 将数据传入模型中，获得输出的预测标签\n",
    "        outputs = model(x)\n",
    "        \n",
    "        # 将预测标签与真实标签计算损失\n",
    "        loss = criterion(outputs, y)\n",
    "        \n",
    "        # 记录当前损失\n",
    "        running_loss += loss.item()\n",
    "        \n",
    "        # 记录当前准确率\n",
    "        _, preds = torch.max(outputs.data, 1)\n",
    "        running_correct += torch.sum(preds == y)\n",
    "        \n",
    "        # 反向传播\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "\n",
    "    loss = running_loss / len(dataloader.dataset)\n",
    "    accuracy = running_correct / len(dataloader.dataset)\n",
    "\n",
    "    print(f\"Train Loss: {loss:.4f}, Train Acc: {accuracy:.4f}\")\n",
    "\n",
    "    return loss, accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义验证函数，我们可以通过验证集对模型的参数进行调整\n",
    "def validate(model, dataloader):\n",
    "\n",
    "    model.eval()\n",
    "    \n",
    "    # 初始化loss以及模型准确率\n",
    "    running_loss = 0.0\n",
    "    correct = 0\n",
    "    with torch.no_grad():\n",
    "        for i, data in enumerate(dataloader):\n",
    "            # 流程同训练\n",
    "            x, y = data[0].to(device), data[1].to(device)\n",
    "            outputs = model(x)\n",
    "            loss = criterion(outputs, y)\n",
    "\n",
    "            running_loss += loss.item()\n",
    "            _, preds = torch.max(outputs.data, 1)\n",
    "            correct += torch.sum(preds == y)\n",
    "            \n",
    "        loss = running_loss / len(dataloader.dataset)\n",
    "        accuracy = correct / len(dataloader.dataset)\n",
    "        print(f'Val Loss: {loss:.4f}, Val Acc: {accuracy:.4f}')\n",
    "\n",
    "        return loss, accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义测试函数，评估模型的效果\n",
    "def test(model, dataloader):\n",
    "    correct = 0\n",
    "    total = 0\n",
    "    with torch.no_grad():\n",
    "        for data in dataloader:\n",
    "            x, y = data[0].to(device), data[1].to(device)\n",
    "            outputs = model(x)\n",
    "            _, predicted = torch.max(outputs.data, 1)\n",
    "            total += y.size(0)\n",
    "            correct += torch.sum(predicted == y)\n",
    "    return correct, total\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们开始训练模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training on 5486 examples, validating on 1829 examples...\n",
      "\n",
      "Epoch 1 of 25\n",
      "Train Loss: 0.0024, Train Acc: 0.0840\n",
      "Val Loss: 0.0019, Val Acc: 0.3100\n",
      "\n",
      "Epoch 2 of 25\n",
      "Train Loss: 0.0017, Train Acc: 0.3828\n",
      "Val Loss: 0.0015, Val Acc: 0.4352\n",
      "\n",
      "Epoch 3 of 25\n",
      "Train Loss: 0.0014, Train Acc: 0.5314\n",
      "Val Loss: 0.0013, Val Acc: 0.5145\n",
      "\n",
      "Epoch 4 of 25\n",
      "Train Loss: 0.0012, Train Acc: 0.6469\n",
      "Val Loss: 0.0011, Val Acc: 0.5894\n",
      "\n",
      "Epoch 5 of 25\n",
      "Train Loss: 0.0009, Train Acc: 0.7211\n",
      "Val Loss: 0.0010, Val Acc: 0.6698\n",
      "\n",
      "Epoch 6 of 25\n",
      "Train Loss: 0.0008, Train Acc: 0.7913\n",
      "Val Loss: 0.0008, Val Acc: 0.7523\n",
      "\n",
      "Epoch 7 of 25\n",
      "Train Loss: 0.0006, Train Acc: 0.8482\n",
      "Val Loss: 0.0007, Val Acc: 0.7922\n",
      "\n",
      "Epoch 8 of 25\n",
      "Train Loss: 0.0005, Train Acc: 0.8941\n",
      "Val Loss: 0.0006, Val Acc: 0.8387\n",
      "\n",
      "Epoch 9 of 25\n",
      "Train Loss: 0.0004, Train Acc: 0.9306\n",
      "Val Loss: 0.0005, Val Acc: 0.8737\n",
      "\n",
      "Epoch 10 of 25\n",
      "Train Loss: 0.0003, Train Acc: 0.9564\n",
      "Val Loss: 0.0004, Val Acc: 0.8961\n",
      "\n",
      "Epoch 11 of 25\n",
      "Train Loss: 0.0002, Train Acc: 0.9750\n",
      "Val Loss: 0.0004, Val Acc: 0.9103\n",
      "\n",
      "Epoch 12 of 25\n",
      "Train Loss: 0.0002, Train Acc: 0.9869\n",
      "Val Loss: 0.0003, Val Acc: 0.9202\n",
      "\n",
      "Epoch 13 of 25\n",
      "Train Loss: 0.0001, Train Acc: 0.9933\n",
      "Val Loss: 0.0003, Val Acc: 0.9328\n",
      "\n",
      "Epoch 14 of 25\n",
      "Train Loss: 0.0001, Train Acc: 0.9958\n",
      "Val Loss: 0.0003, Val Acc: 0.9311\n",
      "\n",
      "Epoch 15 of 25\n",
      "Train Loss: 0.0001, Train Acc: 0.9973\n",
      "Val Loss: 0.0002, Val Acc: 0.9355\n",
      "\n",
      "Epoch 16 of 25\n",
      "Train Loss: 0.0001, Train Acc: 0.9985\n",
      "Val Loss: 0.0002, Val Acc: 0.9360\n",
      "\n",
      "Epoch 17 of 25\n",
      "Train Loss: 0.0000, Train Acc: 0.9989\n",
      "Val Loss: 0.0002, Val Acc: 0.9360\n",
      "\n",
      "Epoch 18 of 25\n",
      "Train Loss: 0.0000, Train Acc: 0.9987\n",
      "Val Loss: 0.0002, Val Acc: 0.9377\n",
      "\n",
      "Epoch 19 of 25\n",
      "Train Loss: 0.0000, Train Acc: 0.9993\n",
      "Val Loss: 0.0002, Val Acc: 0.9399\n",
      "\n",
      "Epoch 20 of 25\n",
      "Train Loss: 0.0000, Train Acc: 0.9996\n",
      "Val Loss: 0.0002, Val Acc: 0.9410\n",
      "\n",
      "Epoch 21 of 25\n",
      "Train Loss: 0.0000, Train Acc: 0.9996\n",
      "Val Loss: 0.0002, Val Acc: 0.9404\n",
      "\n",
      "Epoch 22 of 25\n",
      "Train Loss: 0.0000, Train Acc: 0.9998\n",
      "Val Loss: 0.0002, Val Acc: 0.9415\n",
      "\n",
      "Epoch 23 of 25\n",
      "Train Loss: 0.0000, Train Acc: 0.9998\n",
      "Val Loss: 0.0002, Val Acc: 0.9426\n",
      "\n",
      "Epoch 24 of 25\n",
      "Train Loss: 0.0000, Train Acc: 0.9998\n",
      "Val Loss: 0.0002, Val Acc: 0.9420\n",
      "\n",
      "Epoch 25 of 25\n",
      "Train Loss: 0.0000, Train Acc: 1.0000\n",
      "Val Loss: 0.0002, Val Acc: 0.9420\n"
     ]
    }
   ],
   "source": [
    "# 开始对模型进行训练\n",
    "\n",
    "# 设定迭代的轮次\n",
    "epochs = 25\n",
    "# 设定训练以及验证的损失与准确率\n",
    "train_loss , train_accuracy = [], []\n",
    "val_loss , val_accuracy = [], []\n",
    "\n",
    "print(f\"Training on {len(train_data)} examples, validating on {len(val_data)} examples...\")\n",
    "\n",
    "# 开始迭代\n",
    "for epoch in range(epochs):\n",
    "    # 输出迭代信息\n",
    "    print(f\"\\nEpoch {epoch+1} of {epochs}\")\n",
    "    train_epoch_loss, train_epoch_accuracy = fit(model, trainloader)\n",
    "    val_epoch_loss, val_epoch_accuracy = validate(model, valloader)\n",
    "    train_loss.append(train_epoch_loss)\n",
    "    train_accuracy.append(train_epoch_accuracy.cpu())\n",
    "    val_loss.append(val_epoch_loss)\n",
    "    val_accuracy.append(val_epoch_accuracy.cpu())\n",
    "    \n",
    "# 可以保存训练的模型\n",
    "# torch.save(model.state_dict(), f\"../outputs/models/resnet34_epochs{epochs}.pth\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们绘制并观察准确率和损失曲线。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7ff9ae7ba350>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 41844 characters]",
      "text/plain": [
       "<Figure size 1000x700 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 43128 characters]",
      "text/plain": [
       "<Figure size 1000x700 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 绘制准确率曲线\n",
    "plt.figure(figsize=(10, 7))\n",
    "# 训练集的准确率曲线\n",
    "plt.plot(train_accuracy, color='green', label='train accuracy')\n",
    "# 验证集准确率曲线\n",
    "plt.plot(val_accuracy, color='blue', label='validataion accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "# 绘制loss曲线\n",
    "plt.figure(figsize=(10, 7))\n",
    "plt.plot(train_loss, color='orange', label='train loss')\n",
    "plt.plot(val_loss, color='red', label='validataion loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后，我们对模型的准确率进行评估。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of the network on test images: 93.111 %\n"
     ]
    }
   ],
   "source": [
    "correct, total = test(model, testloader)\n",
    "print('Accuracy of the network on test images: %0.3f %%' % (100 * correct / total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10.5 小结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这一章中，我们学习了视觉识别中最基础的问题——图像分类。我们先后介绍了基于手工设计图像表征（即视觉词袋模型）的图像分类算法和基于深度卷积神经网络的图像分类算法。在后续的章节中，我们将介绍如何把基于深度卷积神经网络的图像分类算法推广到其他的视觉识别问题中。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10.6 习题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 习题10.1：词袋模型中，视觉词典的大小，即视觉单词的个数由超参$K$决定。如果$K$设置得过小或者过大，分别对词袋模型的图像表征能力有何影响？\n",
    "\n",
    "#### 习题10.2：在我们介绍的词袋模型中，对每个图像特征的编码我们采用的是最近邻编码，即将这个特征分配给视觉词典中与其最相似的一个视觉单词。我们也可以采用其他的编码策略，比如我们可以这个特征分配给视觉词典中与其最相似$N$个视觉单词，分配的权重由它们之间的相似度确定。请你动手实现基于这样一个编码策略的词袋模型。\n",
    "\n",
    "#### 习题10.3：在基于深度卷积神经网络的图像分类算法中，我们对图像数据做了很多增强。请你通过实验验证数据增强对分类结果的影响。\n",
    "\n",
    "#### 习题10.4：在基于深度卷积神经网络的图像分类算法中，我们对网络模型进行了预训练。请你通过实验验证预训练对分类结果的影响。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 参考文献"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1] Svetlana Lazebnik, Cordelia Schmid, Jean Ponce: Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories. CVPR (2) 2006: 2169-2178\n",
    "\n",
    "[2] Josef Sivic, Andrew Zisserman: Video Google: A Text Retrieval Approach to Object Matching in Videos. ICCV 2003: 1470-1477\n",
    "\n",
    "[3] Stephen O'Hara, Bruce A. Draper: Introduction to the Bag of Features Paradigm for Image Classification and Retrieval. CoRR abs/1101.3354 (2011)\n",
    "\n",
    "[4] Tsai C F. Bag-of-words representation in image annotation: A review. International Scholarly Research Notices, 2012\n",
    "\n",
    "[5] Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, Lawrence D. Jackel:\n",
    "Backpropagation Applied to Handwritten Zip Code Recognition. Neural Comput. 1(4): 541-551 (1989)\n",
    "\n",
    "[6] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner: Gradient-based learning applied to document recognition. Proc. IEEE 86(11): 2278-2324 (1998)\n",
    "\n",
    "[7] Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton: ImageNet Classification with Deep Convolutional Neural Networks. NIPS 2012: 1106-1114\n",
    "\n",
    "[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun: Deep Residual Learning for Image Recognition. CVPR 2016: 770-778\n",
    "\n",
    "[9]Yann LeCun. The MNIST database of handwritten digits[J]. http://yann.lecun.com/exdb/mnist/, 1998.\n",
    "\n",
    "[10] Li Fei-Fei, Rob Fergus, Pietro Perona: Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories. CVPR Workshops 2004: 178\n",
    "\n",
    "[11] Krizhevsky A, Hinton G. Learning multiple layers of features from tiny images. 2009.\n",
    "\n",
    "[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei: ImageNet: A large-scale hierarchical image database. CVPR 2009: 248-255\n",
    "\n",
    "[13] C. Liu, J. Yuen, A. Torralba, J. Sivic, W. Freeman, Sift flow: dense correspondence across different scenes, in: Proceedings of 10th European Conference on Computer Vision, 2008, pp. 28–42.\n",
    "\n",
    "[14] C. Liu, J. Yuen, A. Torralba, Sift flow: dense correspondence across scenes and its applications\n",
    "IEEE Trans. Pattern Anal. Mach. Intell., 33 (5) (2011), pp. 978-994"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
