{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 11. 语义分割"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.1 简介"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "语义分割是计算机视觉中的一个基础核心问题，也是一项具有挑战性的视觉识别任务。简单来说，语义分割的目标是为一张图像的每个像素预测一个类别标签，即它是一种像素级别的“图像分类”，如图 11-1 所示。需要注意的是，与第9章中所介绍的无监督图像分割有所不同，语义分割得到的分割是具有语义性的，每个分割对应于给定的标注集中的一个类别。如图 11-2 所示，其中粉红色和玫红色的分割分别对应于“人”和“马”这两种类别。用数学语言来描述语义分割的问题即是：定义图像空间$\\mathcal{I}$和预定义的类别集合$\\mathcal{C}$。给定数据集$\\mathcal{D} = \\{(\\mathbf{I}^{(n)},\\mathbf{Y}^{(n)})\\}_{n=1}^N$，其中$\\mathbf{I}^{(n)}\\in \\mathbb{I}$是数据集中的第$n$张图像，$\\mathbf{Y}^{(n)}$是这张图像对应的类别标签图（label map），类别标签图中的第$i$个元素$y_i^{(n)}\\in\\mathcal{C}$是图像中第$i$个像素的类别标签；语义分割的任务是从$\\mathcal{D}$中学习得到一个从图像空间到类别标签图空间的映射$f:\\mathbb{I} \\rightarrow \\mathbb{Y}$，其中$\\forall n, \\mathbf{Y}^{(n)} \\in \\mathbb{Y}$，从而给定任意一张测试图像$\\mathbf{I}$，我们可以用学习得到的映射函数$f$预测其标签：$\\hat{\\mathbf{Y}}=f(\\mathbf{I})$。与图像分类类似，这个学习过程需要建模条件概率$p(\\mathbf{Y}|\\mathbf{I})$，则$\\hat{\\mathbf{Y}}=f(\\mathbf{I})=\\arg\\max_{\\mathbf{Y}\\in\\mathcal{Y}}p(\\mathbf{Y}|\\mathbf{I})$。语义分割能够提供像素级别的类别信息，因此在实际场景中有很多应用，如自动驾驶、缺陷检测、医学影像辅助诊断等。\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic2.zhimg.com/80/v2-9b9ad2e16e149145c802939acc1e2619_1440w.jpg\n",
    "\" width=600>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图11-1 语义分割是一种像素级的分类任务。</div>\n",
    "</center>\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic3.zhimg.com/80/v2-d048c6ecd2d0299bc6e7c9de071d9b0a_1440w.jpg\n",
    "\" width=300>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图11-2 语义分割示例。</div>\n",
    "</center>\n",
    "\n",
    "在这一章中，我们将在基于深度学习的图像分类的基础上实现语义分割。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.2 数据集和评测指标"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用于语义分割评测的标准数据集包括：\n",
    "\n",
    "1. Pascal Voc [1]：Pascal Voc数据集是PASCAL VOC挑战官方使用的数据集。该数据集包含20类共1万张的图像，每张图像都有完整的标签。\n",
    "2. Cityscapes [2]：Cityscapes数据集采集自德国及附近国家的50个城市，包括了春夏秋三个季节的街区场景，包含了5千张图像共19个类别，每张图像都有完整的标签。\n",
    "3. ADE20K [3]：ADE20K数据集是MIT发布的场景理解数据集，包含2万张标记完整的图像共150个类别。\n",
    "\n",
    "在语义分割中，我们通常采用平均交并比（mean Intersection over Union, mIoU）作为评测指标。令$n_{cl}$表示本属于类别$c\\in\\mathcal{C}$但被预测为类别$l\\in\\mathcal{C}$的像素点的总数，令$t_c=\\sum_l n_{cl}$表示所有属于类别$c$的像素点总数，则平均交并比mIoU定义如下：\n",
    "\n",
    "$$\\texttt{mIoU} = \\frac{1}{|\\mathcal{C}|} \\sum_{c\\in\\mathcal{C}} \\frac{n_{cc}}{t_c + \\sum_{l\\in\\mathcal{C}} n_{lc} -n_{cc}}$$\n",
    "其中$|\\cdot|$表示集合中元素的个数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.3 基于深度学习的语义分割模型——全卷积神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们知道语义分割实际上是进行逐像素的分类，因此，一个最为直观的语义分割方法便是训练一个可以对每个像素进行分类的网络。如图 11-3 所示，我们以图像的每一个像素点为中心提取一个“图像块”（patch），并为其赋予该中心像素对应的类别标签。接着，我们训练一个CNN模型用于对这些图像块进行分类。在对图像上的每一个像素点进行类别预测之后，我们便得到了语义分割的结果，如图 11-3 最右侧图所示。\n",
    "\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic3.zhimg.com/80/v2-020b2e2159e7f509cb67482447142116_1440w.jpg\n",
    "\" width=800>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图11-3 一个直观的语义分割模型。</div>\n",
    "</center>\n",
    "\n",
    "上述的方法虽然直观，但实际中却存在许多的问题：1. 由于一张图像中可能存在成千上万个像素点，且相邻像素点的图像块有大部分的重叠，使得这一方法计算量大、计算效率低。2. 与整张图像相比，图像块显得比较小，而且只能提取图像局部的特征，这使得分类的性能受限。3. 整个过程不是端到端的，它需要有预处理和后处理的环节。为了应对这些问题，2015年Jonathan Long等人在此基础上提出了一种全新的分割网络——全卷积神经网络（Fully Convolutional Network，FCN) [4]。与传统的CNN网络（如AlexNet等）不同的是，在FCN中，网络输出的全连接层被替换为一个卷积核大小为$1\\times1$的卷积层，如图 11-4 所示。根据上一章的内容，对于图像分类，我们知道可以使用全连接层将卷积层输出的多通道图像特征图映射成一个维度为$|\\mathcal{C}|$的向量，该向量中每个元素的值表示属于对应类别的置信度。如此我们便可以实现一张图像的分类。然而在语义分割中，我们需要对一张图像的每一个像素点进行分类，因此全连接层并不适用。为了实现端到端的预测，我们使用一个$1\\times1$的卷积层替换全连接层（为什么$1\\times1$的卷积层可以替换全连接层？），将卷积层输出的多通道特征图映射成一个空间分辨率不变的通道数量为$|\\mathcal{C}|$的类别置信度图。这样，卷积层输出的特征图上的每一个空间位置都对应一个维度为$|\\mathcal{C}|$的向量，表示该位置对应的原始图像区域属于$\\mathcal{C}$中每个类别的置信度，如此便可实现图像的语义分割。我们把类别置信度图叫做“热度图”（heat map），属于某类置信度越高，则热度越高。调整热度图的空间分辨率，使其和原图一致，便可以得到我们所需要的语义分割的结果。不过，在图 11-4 中我们依旧可以发现两个明显的问题：随着卷积的进行，特征图也越来越小，最终输出的热度图要比原图小很多；由于最终输出的特征图较小，其蕴含的边界信息也变得十分的粗糙，无法精准地勾勒出原图中物体的轮廓。为了解决上述两个问题，Jonathan Long等人在FCN网络中引入两种有效的架构：上采样（Unsampling）和跳跃连接（Skip-connection）。\n",
    "\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic1.zhimg.com/80/v2-d7ed66df02bd43d859639dc0fb76adf0_1440w.jpg\n",
    "\" width=600>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图11-4 用$1\\times1$卷积层替换所有的全连接层。</div>\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.3.1 上采样"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "为解决输出的热度图分辨率小于输入图像的问题，Jonathan Long等人 提出对特征图进行上采样，使其和原始图像的空间分辨率保持一致。上采样一般可以通过双线性插值法实现。如图 11-5 所示，设$g$是一个二维信号，若已知$g$在点$Q_{11}=(x_{1}, y_{1}), Q_{12}=(x_{1}, y_{2}), Q_{21}=(x_{2}, y_{1}), Q_{22}=(x_{2}, y_{2})$处的值，需要求$g$在点$P=(x,y)$处的值。求值的过程实际上就是双线性插值的过程。我们重点关注点$P$在$x$方向和$y$方向上与其余四个点的距离。我们首先先在$x$方向上进行插值：\n",
    "$$g(x, y_{1}) \\approx \\frac{x-x_{1}}{x_{2}-x_{1}}[g(Q_{21})-g(Q_{11})]+g(Q_{11})=\\frac{x_{2}-x}{x_{2}-x_{1}} g(Q_{11})+\\frac{x-x_{1}}{x_{2}-x_{1}} g(Q_{21}) \\\\ g(x, y_{2}) \\approx \\frac{x-x_{1}}{x_{2}-x_{1}}[g(Q_{22})-g(Q_{12})]+g(Q_{12}) =\\frac{x_{2}-x}{x_{2}-x_{1}} g(Q_{12})+\\frac{x-x_{1}}{x_{2}-x_{1}} g(Q_{22})\n",
    "$$\n",
    "\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic2.zhimg.com/80/v2-66432863549dccc82a5ac033cbb1e035_1440w.webp\n",
    "\" width=200>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图11-5 双线性插值法举例。</div>\n",
    "</center>\n",
    "\n",
    "接着，我们在$y$方向上进行插值：\n",
    "\n",
    "$$g(x, y)  \\approx g(x, y_{1})+\\frac{y-y_{1}}{y_{2}-y_{1}}[g(x, y_{2})-g(x, y_{1})] \n",
    "=\\frac{y_{2}-y}{y_{2}-y_{1}} g(x, y_{1})+\\frac{y-y_{1}}{y_{2}-y_{1}} g(x, y_{2}) \n",
    "=\\frac{1}{(x_{2}-x_{1})(y_{2}-y_{1})}[\\begin{array}{ll}x_{2}-x & x-x_{1}\\end{array}] \\left[\\begin{array}{ll} g(Q_{11}) & g(Q_{12}) \\\\ g(Q_{21}) & g(Q_{22})\\end{array}\\right]\\left[\\begin{array}{l} y_{2}-y \\\\y-y_{1}\\end{array}\\right]\n",
    "$$\n",
    "\n",
    "\n",
    "如此，我们便可以求出点$P$处的值。可以发现，双线性插值其实就是在两个方向上分别进行插值，而每一个方向“插值”的过程实际上类似于计算一条直线在某一点函数值的过程。当我们学会了双线性插值之后，如图 11-6 所示，我们便可以通过对一张分辨率为$2\\times 2$的图像上采样将其分辨率变为$4\\times 4$。\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic1.zhimg.com/80/v2-1f0142501d3ba76e85913e0a68f599e4_1440w.jpg\n",
    "\" width=400>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图11-6 对一张分辨率为$2\\times 2$的图像上采样，将图像大小变为$4\\times 4$。</div>\n",
    "</center>\n",
    "\n",
    "观察上述数学形式，可以发现双线性插值可以通过卷积实现。所以我们可以用深度网络中的卷积层实现特征图的上采样，只是这个卷积层的卷积核是固定的。那么是否可以用可学习的卷积核来实现特征图上采样呢？当然也是可以的。这种可以实现特征图上采样的可学习卷积称为“转置卷积”或“反卷积”，可以通过一个步长小于1的卷积操作来实现。实际上，当卷积的步长$s<1$时，对图像进行卷积，相当于在图像的相邻像素点之间插入$[1/s-1]$个像素点之后，再进行步长为1的卷积，其中$[\\cdot]$表示取整函数。如图 11-7 所示，我们对输入图像做步长为0.5的卷积实现对其的上采样：首先需要在图像的每两个相邻像素之间插入值为零的像素点。接着，对其做用步长为1的卷积，卷积过程如图 11-7 所示。同时，对于输入的图像，我们也可以在其周围进行补零，最终将图像从$2\\times2$上采样到$5\\times5$。在实际操作中，我们对特征图的每个通道依次进行上采样，得到最终的结果。\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic4.zhimg.com/80/v2-56ea550ab87b004b4a2f23ab157fea1b_1440w.jpg\n",
    "\" width=400>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图11-7 通过反卷积的方式实现上采样。</div>\n",
    "</center>\n",
    "\n",
    "在代码中，我们通过调用以下函数实现反卷积："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ConvTranspose2d(3, 3, kernel_size=(3, 3), stride=(2, 2))"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "torch.nn.ConvTranspose2d(in_channels=3,         # 输入数据的通道数\n",
    "                        out_channels=3,         # 输出数据的通道数\n",
    "                        kernel_size=3,          # 卷积核的大小\n",
    "                        stride=2,               # 卷积步长\n",
    "                        padding=0,              # 原图周围需要填充的格子行（列）数\n",
    "                        output_padding=0,       # 输出特征图边缘需要填充的行（列）数，一般不设置\n",
    "                        groups=1,               # 分组卷积的组数，一般默认设置为1\n",
    "                        bias=True               # 卷积偏置，一般设置为False，True的话可以增加模型的泛化能力\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于是反卷积，因此实际进行的步长为$1/stride$。同时，在实际操作中，卷积核的参数可以通过学习得到，也可以利用双线性插值的原理将其固定。利用双线性插值固定卷积核权重的代码如下所示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 利用双线性插值法设置卷积核权重\n",
    "def bilinear_kernel(in_channels, out_channels, kernel_size):\n",
    "    \n",
    "    # 找到卷积核正中心的位置\n",
    "    factor = (kernel_size + 1) // 2\n",
    "    if kernel_size % 2 == 1:\n",
    "        center = factor - 1\n",
    "    else:\n",
    "        center = factor - 0.5\n",
    "    # np.ogrid()函数构建一个二维矩阵\n",
    "    og = np.ogrid[:kernel_size, :kernel_size]\n",
    "    print('og', og)\n",
    "    \n",
    "    # 构建卷积核权重\n",
    "    filt = (1 - abs(og[0] - center) / factor) * (1 - abs(og[1] - center) / factor)\n",
    "    print('filt', filt)\n",
    "    \n",
    "    weight = np.zeros((in_channels,out_channels,kernel_size,kernel_size), dtype='float32')\n",
    "    weight[range(in_channels), range(out_channels),:,:]=filt\n",
    "    \n",
    "    return torch.from_numpy(weight)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们以一个简单的例子进一步说明上述代码块中的部分变量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "og [array([[0],\n",
      "       [1],\n",
      "       [2]]), array([[0, 1, 2]])]\n",
      "filt [[0.25 0.5  0.25]\n",
      " [0.5  1.   0.5 ]\n",
      " [0.25 0.5  0.25]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[[[0.2500, 0.5000, 0.2500],\n",
       "          [0.5000, 1.0000, 0.5000],\n",
       "          [0.2500, 0.5000, 0.2500]]]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "bilinear_kernel(1, 1, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.3.2 跳跃连接"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic4.zhimg.com/80/v2-da4000ce2e564398afe9faafc107e0b7_1440w.jpg\n",
    "\" width=800>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图11-8 融合中间卷积层的特征，这些特征包含了局部与全局的信息，也包含了深层的与浅层的信息。</div>\n",
    "</center>\n",
    "\n",
    "由于主干网络最后一层输出的特征图空间分辨率太低（如图 11-4），上采样之后预测得到的类别标签图会很粗糙，无法精准地勾勒出原图中物体的轮廓。一般情况而言，浅层卷积层的特征图较大，能够精确定位物体的边界，但是语义性不够；深层卷积层的特征图较小，但是语义性强。那么有没有什么办法在得到较大的特征图的同时，也能使特征图拥有丰富的语义信息呢？为此，Jonathan Long等人提出可以通过跳跃连接融合网络中不同层次的卷积层特征以解决该问题。如图 11-8 所示，我们可以将中间不同层级的特征图连接起来，连接后的特征图包含了全局的与局部的信息，也包含了浅层的与深层的信息。通过跳跃连接，构建既具有很强语义性，又包含精确物体位置信息的特征图，从而实现精准的语义分割。\n",
    "\n",
    "\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://img-blog.csdnimg.cn/9a021d7bf5d94c97a3c1741939a65651.png#pic_center\n",
    "\" width=700>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图11-9 FCN中的跳跃连接</div>\n",
    "</center>\n",
    "\n",
    "\n",
    "具体而言，如图 11-9 所示，从FCN的主干网络可以得到5个不同空间分辨率的特征图，它们的空间分辨率分别是原始图像的1/2，1/4，1/8，1/16和1/32。连接特征图的方式有多种，根据连接的特征图层次的不同，FCN有不同的变种，包括FCN-8s，FCN-16s和FCN-32s。由于不同层次的特征图不但空间分辨率不一样，它们的通道数也不一样。所以在对它们进行连接时，需要先将它们的通道数调整成一致。如何在不改变特征图的空间分辨率的情况下改变特征图的通道数？没错，我们刚刚介绍过的$1 \\times 1$卷积正好可以实现这个功能。我们用$1 \\times 1$卷积将不同层次的特征图的通道数都调整为$|\\mathcal{C}|$。对于主干网络pool5层输出的特征图，在通过$1 \\times 1$卷积调整完通道数后，直接对其上采样32倍得到与原始图像相同空间分辨率的特征图，这样的模型被称为FCN-32s；我们也可以将主干网络pool5输出的特征图两倍上采样后，与主干网络pool4输出的特征图进行逐元素相加（element-wise sum），再将相加后的特征图进行16倍的上采样得到与原始图像相同空间分辨率的特征图，这样的模型被称为FCN-16s；同样，我们还可以将上述相加后的特征图先进行2倍上采样，然后与主干网络pool3输出的特征图逐元素相加得到新的特征图，再将这个特征图进行8倍上采样得到与原始图像相同空间分辨率的特征图，这样的模型被称为FCN-8s。\n",
    "\n",
    "为了更好的说明这一过程，我们来看FCN8s的代码实现。这里，我们选择ResNet101作为模型的主干网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision import models\n",
    "\n",
    "# 使用 ResNet101 作为主干网络，模型使用ImageNet预训练\n",
    "pretrained_net = models.resnet101(pretrained='imagenet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "# FCN-8s模型\n",
    "class FCN8s(nn.Module):\n",
    "    def __init__(self, num_classes):\n",
    "        super(FCN8s, self).__init__()\n",
    "        # 这里我们使用children()调用ResNet101的部分网络，该深度特征图的大小为输入图像的1/8\n",
    "        self.stage1 = nn.Sequential(*list(pretrained_net.children())[:-4])\n",
    "        # 该深度特征图的大小为输入图像的1/16\n",
    "        self.stage2 = list(pretrained_net.children())[-4]\n",
    "        # 该深度特征图的大小为输入图像的1/32\n",
    "        self.stage3 = list(pretrained_net.children())[-3]\n",
    "        \n",
    "        # 调整stage3输出的通道数\n",
    "        self.scores1 = nn.Conv2d(2048, num_classes, 1)\n",
    "        # 调整stage2输出的通道数\n",
    "        self.scores2 = nn.Conv2d(1024, num_classes, 1)\n",
    "        # 调整stage1输出的通道数\n",
    "        self.scores3 = nn.Conv2d(512, num_classes, 1)\n",
    "        \n",
    "        # 对pool3与pool4、pool5的输出进行8倍上采样\n",
    "        self.upsample_8x = nn.ConvTranspose2d(num_classes, num_classes, 16, 8, 4, bias=False)\n",
    "        self.upsample_8x.weight.data = bilinear_kernel(num_classes, num_classes, 16) # 使用双线性 kernel\n",
    "        \n",
    "        # 对pool4和pool5融合的特征2倍上采样\n",
    "        self.upsample_4x = nn.ConvTranspose2d(num_classes, num_classes, 4, 2, 1, bias=False)\n",
    "        self.upsample_4x.weight.data = bilinear_kernel(num_classes, num_classes, 4) # 使用双线性 kernel\n",
    "        \n",
    "        # 对pool5的输出2倍上采样\n",
    "        self.upsample_2x = nn.ConvTranspose2d(num_classes, num_classes, 4, 2, 1, bias=False)   \n",
    "        self.upsample_2x.weight.data = bilinear_kernel(num_classes, num_classes, 4) # 使用双线性 kernel\n",
    "\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.stage1(x)\n",
    "        # s1 1/8\n",
    "        s1 = x\n",
    "        \n",
    "        x = self.stage2(x)\n",
    "        # s2 1/16\n",
    "        s2 = x\n",
    "\n",
    "        x = self.stage3(x)\n",
    "        # s3 1/32\n",
    "        s3 = x\n",
    "                \n",
    "        # 调整pool5输出特征图的通道数\n",
    "        s3 = self.scores1(s3)\n",
    "        # 进行两倍上采样\n",
    "        s3 = self.upsample_2x(s3)\n",
    "        \n",
    "        # 调整pool4输出特征图的通道数\n",
    "        s2 = self.scores2(s2)\n",
    "        # 融合pool5、pool4的特征图\n",
    "        s2 = s2 + s3\n",
    "        \n",
    "        # 调整pool3输出特征图的通道数\n",
    "        s1 = self.scores3(s1)\n",
    "        # 将s2两倍上采样\n",
    "        s2 = self.upsample_4x(s2)\n",
    "        # 融合特征图\n",
    "        s = s1 + s2\n",
    "        \n",
    "        # 8倍上采样得到与原图像大小一致的特征图\n",
    "        s = self.upsample_8x(s2)\n",
    "        return s\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不知道通过代码，你对FCN的理解是不是更深刻了呢？通过上采样得到的特征图具有和原始图像相同的空间分辨率，基于该特征图可以进行逐像素的类别预测，这样可以逐像素计算预测类别标签和真实类别标签之间的交叉熵损失函数以训练FCN：\n",
    "\n",
    "$$\\ell(\\mathcal{D}) = -\\frac{1}{N}\\sum_{n=1}^N\\sum_i \\log(p(y^{(n)}_i|\\mathbf{I}^{(n)}))$$\n",
    "\n",
    "其中$p(y^{(n)}_i|\\mathbf{I}^{(n)})$表示训练数据集$\\mathcal{D}$中第$n$张图像熵的第$i$个像素对应的类别标签被FCN预测为$y^{(n)}_i\\in\\mathcal{C}$的概率。\n",
    "\n",
    "\n",
    "我们现在总结一下用FCN进行语义分割的流程：首先利用主干网络对图像进行特征提取，主干网络的不同层次可以输出空间分辨率不同的特征图；然后用$1\\times 1$的卷积层将不同层次的特征图的通道数调整为$|\\mathcal{C}|$；接着用跳跃连接融合不同层次的特征图，并通过上采样将特征图的空间分辨率增大到与原始图像一致；最后基于上采样后的特征图进行逐像素的类别预测。FCN进行语义分割的整体流程框架如图 11-10 所示。\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic3.zhimg.com/80/v2-b6471052837da9adfa436a6a9bdf4fbe_1440w.jpg\n",
    "\" width=600>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图11-10 FCN进行语义分割的整体流程框架。</div>\n",
    "</center>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们将动手编写FCN。我们先导入所需要的库。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.4 FCN代码实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-22T06:50:45.953022Z",
     "iopub.status.busy": "2022-11-22T06:50:45.952484Z",
     "iopub.status.idle": "2022-11-22T06:50:48.729717Z",
     "shell.execute_reply": "2022-11-22T06:50:48.727912Z",
     "shell.execute_reply.started": "2022-11-22T06:50:45.952908Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "import cv2\n",
    "import d2l.torch as d2l\n",
    "from PIL import Image\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.utils.data import Dataset\n",
    "from torch.autograd import Variable\n",
    "from torchvision.transforms import transforms as tfs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据原论文，我们使用Pascal Voc作为实验的数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 数据集地址\n",
    "voc_root = '../../../../dataset_share_ssd/voc/VOCdevkit/VOC2012'\n",
    "\n",
    "\n",
    "# 定义Pascal Voc类\n",
    "class VOCSegDataset(Dataset):\n",
    "    def __init__(self, train, crop_size, transforms):\n",
    "        # 定义数据大小\n",
    "        self.crop_size = crop_size\n",
    "        # 定义数据增强类型\n",
    "        self.transforms = transforms\n",
    "        \n",
    "        # 数据以及对应标签的读取\n",
    "        data_list, label_list = read_images(voc_root, train=train)\n",
    "        self.data_list = self._filter(data_list)\n",
    "        self.label_list = self._filter(label_list)\n",
    "        print('Read'+str(len(self.data_list))+'images')\n",
    "        \n",
    "    def _filter(self, images):\n",
    "        return [im for im in images if (Image.open(im).size[1] >= self.crop_size[0] and\n",
    "                                      Image.open(im).size[0] >= self.crop_size[1])]\n",
    "    # 定义数据如何进行传输\n",
    "    def __getitem__(self,idx):\n",
    "        img = self.data_list[idx]\n",
    "        label = self.label_list[idx]\n",
    "        # 读取图像\n",
    "        img = cv2.imread(img)\n",
    "        # 将图像通道从BGR转换成RGB\n",
    "        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
    "        \n",
    "        label = cv2.imread(label)\n",
    "        # 因为标签与像素点一一对应，因此也需要将其进行转换\n",
    "        label = cv2.cvtColor(label, cv2.COLOR_BGR2RGB)\n",
    "        # 进行数据增强\n",
    "        img,label = self.transforms(img, label, self.crop_size)\n",
    "        return img, label\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.data_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们对上述类中出现的部分函数进行实现。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-22T06:51:17.650295Z",
     "iopub.status.busy": "2022-11-22T06:51:17.649804Z",
     "iopub.status.idle": "2022-11-22T06:51:17.695748Z",
     "shell.execute_reply": "2022-11-22T06:51:17.694813Z",
     "shell.execute_reply.started": "2022-11-22T06:51:17.650255Z"
    }
   },
   "outputs": [],
   "source": [
    "# 对上述数据集类的方法进行实现\n",
    "\n",
    "# 读取数据集\n",
    "def read_images(root, train=True):\n",
    "    txt_filename = root + \"/ImageSets/Segmentation/\" + ('train.txt' if train else 'val.txt')\n",
    "    with open(txt_filename, 'r') as f:\n",
    "        images = f.read().split()\n",
    "    data = [os.path.join(root, 'JPEGImages', i + '.jpg') for i in images]\n",
    "    label = [os.path.join(root, 'SegmentationClass', i+'.png') for i in images]\n",
    "    return data, label\n",
    "\n",
    "\n",
    "# 进行随机裁剪\n",
    "def rand_crop(data, label, height,width):\n",
    "    h, w, _ = data.shape\n",
    "    top = random.randint(0, h - height)\n",
    "    left = random.randint(0, w - width)\n",
    "    # 裁剪数据\n",
    "    data = data[top:top + height, left:left + width]\n",
    "    # 裁剪标签\n",
    "    label = label[top:top + height, left:left + width]\n",
    "    return data, label\n",
    "\n",
    "\n",
    "# 为图像增强时的像素点建立对应的标签\n",
    "def image2label(im):\n",
    "    data = np.array(im, dtype='int32')\n",
    "    idx = (data[:,:,0] * 256 + data[:,:,1]) * 256 + data[:,:,2]\n",
    "    return np.array(cm2lbl[idx], dtype='int64')\n",
    "\n",
    "\n",
    "# 定义数据增强方法\n",
    "def img_transforms(im, label, crop_size):\n",
    "    im,label = rand_crop(im, label, *crop_size)\n",
    "    im_tfs = tfs.Compose([\n",
    "        tfs.ToTensor(),\n",
    "        tfs.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n",
    "    ])\n",
    "    im = im_tfs(im)\n",
    "    label = image2label(label)\n",
    "    label = torch.from_numpy(label)\n",
    "    return im, label\n",
    "\n",
    "\n",
    "\n",
    "# 删除双线性插值函数中的输出\n",
    "def bilinear_kernel(in_channels, out_channels, kernel_size):\n",
    "    factor = (kernel_size + 1) // 2\n",
    "    if kernel_size % 2 == 1:\n",
    "        center = factor - 1\n",
    "    else:\n",
    "        center = factor - 0.5\n",
    "    og = np.ogrid[:kernel_size, :kernel_size]\n",
    "    filt = (1 - abs(og[0] - center) / factor) * (1 - abs(og[1] - center) / factor)\n",
    "    weight = np.zeros((in_channels,out_channels,kernel_size,kernel_size), dtype='float32')\n",
    "    weight[range(in_channels), range(out_channels),:,:]=filt\n",
    "    return torch.from_numpy(weight)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着，我们定义一些需要用到的全局变量，如类别信息等，并开始加载数据集，构建数据加载器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-22T06:51:25.202586Z",
     "iopub.status.busy": "2022-11-22T06:51:25.202182Z",
     "iopub.status.idle": "2022-11-22T06:52:07.293551Z",
     "shell.execute_reply": "2022-11-22T06:52:07.292431Z",
     "shell.execute_reply.started": "2022-11-22T06:51:25.202553Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read1114images\n",
      "Read1078images\n"
     ]
    }
   ],
   "source": [
    "# Pascal Voc的类别\n",
    "classes = ['background','aeroplane','bicycle','bird','boat',\n",
    "           'bottle','bus','car','cat','chair','cow','diningtable',\n",
    "           'dog','horse','motorbike','person','potted plant',\n",
    "           'sheep','sofa','train','tv/monitor']\n",
    "\n",
    "# colormap的数量与类别数对应，每一种类别都有其独一无二的颜色，便于绘制图像时进行观察\n",
    "colormap = [[0,0,0], [128,0,0], [0,128,0], [128,128,0], [0,0,128],\n",
    "            [128,0,128], [0,128,128], [128,128,128], [64,0,0], [192,0,0],\n",
    "            [64,128,0], [192,128,0], [64,0,128], [192,0,128],\n",
    "            [64,128,128], [192,128,128], [0,64,0], [128,64,0],\n",
    "            [0,192,0], [128,192,0], [0,64,128]]\n",
    "\n",
    "# 读取数据和标签\n",
    "data, label = read_images(voc_root)\n",
    "num_classes = len(classes)\n",
    "\n",
    "# 将colormap转换成类别\n",
    "cm2lbl = np.zeros(256**3)\n",
    "for i, cm in enumerate(colormap):\n",
    "    cm2lbl[(cm[0]*256+cm[1])*256+cm[2]]=i\n",
    "\n",
    "# 设置输入图像大小\n",
    "input_shape = (320,480)\n",
    "voc_train = VOCSegDataset(True, input_shape, img_transforms)\n",
    "voc_test = VOCSegDataset(False, input_shape, img_transforms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#设置batch_size，生成数据加载器\n",
    "BATCH_SIZE = 64\n",
    "train_data = DataLoader(voc_train, batch_size=BATCH_SIZE, shuffle=True)\n",
    "test_data = DataLoader(voc_test, batch_size=BATCH_SIZE)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们先展示数据集中的部分图像与其对应的标签。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-22T06:52:11.770267Z",
     "iopub.status.busy": "2022-11-22T06:52:11.769852Z",
     "iopub.status.idle": "2022-11-22T06:52:12.587582Z",
     "shell.execute_reply": "2022-11-22T06:52:12.586360Z",
     "shell.execute_reply.started": "2022-11-22T06:52:11.770231Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 185268 characters]",
      "text/plain": [
       "<Figure size 1000x1000 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 198820 characters]",
      "text/plain": [
       "<Figure size 1000x1000 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for i, (img, label) in enumerate(voc_train):\n",
    "    plt.figure(figsize=(10, 10))\n",
    "    plt.subplot(221)\n",
    "    plt.imshow(img.moveaxis(0, 2))\n",
    "    plt.subplot(222)\n",
    "    plt.imshow(label)  \n",
    "    plt.show()\n",
    "    plt.close()\n",
    "    if i ==1:\n",
    "        break\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着，我们编写相关函数实现计算mIoU。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _fast_hist(label_true, label_pred, n_class):\n",
    "    mask = (label_true >= 0) & (label_true < n_class)\n",
    "    hist = np.bincount(\n",
    "        n_class * label_true[mask].astype(int) +\n",
    "        label_pred[mask], minlength=n_class ** 2).reshape(n_class, n_class)\n",
    "    return hist\n",
    "\n",
    "\n",
    "def mIoU(label_trues, label_preds, n_class):\n",
    "    # 计算mIoU\n",
    "    hist = np.zeros((n_class, n_class))\n",
    "    \n",
    "    for lt, lp in zip(label_trues, label_preds):\n",
    "        hist += _fast_hist(lt.flatten(), lp.flatten(), n_class)\n",
    "        \n",
    "    iu = np.diag(hist) / (hist.sum(axis=1) + hist.sum(axis=0) - np.diag(hist))\n",
    "    mean_iu = np.nanmean(iu)\n",
    "\n",
    "    return mean_iu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们加载FCN8s模型，并对其进行训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "# 加载模型 \n",
    "model = FCN8s(num_classes)\n",
    "model.to(device)\n",
    "model = nn.DataParallel(model)\n",
    "\n",
    "# 设置优化器\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=1e-2, weight_decay=1e-4)\n",
    "\n",
    "# 设置损失函数\n",
    "criterion = nn.NLLLoss()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0, Train Loss: 1.58653, Train Mean IU: 0.29587,                 Valid Loss: 1.19245, Valid Mean IU: 0.31969 \n",
      "Epoch: 1, Train Loss: 1.06124, Train Mean IU: 0.32866,                 Valid Loss: 0.95682, Valid Mean IU: 0.34103 \n",
      "Epoch: 2, Train Loss: 0.90607, Train Mean IU: 0.35688,                 Valid Loss: 0.82865, Valid Mean IU: 0.37034 \n",
      "Epoch: 3, Train Loss: 0.79397, Train Mean IU: 0.38886,                 Valid Loss: 0.73156, Valid Mean IU: 0.41331 \n",
      "Epoch: 4, Train Loss: 0.70670, Train Mean IU: 0.41854,                 Valid Loss: 0.65698, Valid Mean IU: 0.43930 \n",
      "Epoch: 5, Train Loss: 0.64498, Train Mean IU: 0.44337,                 Valid Loss: 0.60177, Valid Mean IU: 0.46721 \n",
      "Epoch: 6, Train Loss: 0.60861, Train Mean IU: 0.45604,                 Valid Loss: 0.56447, Valid Mean IU: 0.48623 \n",
      "Epoch: 7, Train Loss: 0.56338, Train Mean IU: 0.46788,                 Valid Loss: 0.53989, Valid Mean IU: 0.49255 \n",
      "Epoch: 8, Train Loss: 0.52178, Train Mean IU: 0.49065,                 Valid Loss: 0.50809, Valid Mean IU: 0.50251 \n",
      "Epoch: 9, Train Loss: 0.50536, Train Mean IU: 0.49256,                 Valid Loss: 0.49549, Valid Mean IU: 0.50534 \n",
      "Epoch: 10, Train Loss: 0.49491, Train Mean IU: 0.49236,                 Valid Loss: 0.47784, Valid Mean IU: 0.51250 \n",
      "Epoch: 11, Train Loss: 0.48554, Train Mean IU: 0.49714,                 Valid Loss: 0.47987, Valid Mean IU: 0.50560 \n",
      "Epoch: 12, Train Loss: 0.44740, Train Mean IU: 0.51092,                 Valid Loss: 0.44752, Valid Mean IU: 0.52313 \n",
      "Epoch: 13, Train Loss: 0.43670, Train Mean IU: 0.51541,                 Valid Loss: 0.43500, Valid Mean IU: 0.52710 \n",
      "Epoch: 14, Train Loss: 0.42248, Train Mean IU: 0.51456,                 Valid Loss: 0.42750, Valid Mean IU: 0.52895 \n",
      "Epoch: 15, Train Loss: 0.41235, Train Mean IU: 0.52426,                 Valid Loss: 0.41964, Valid Mean IU: 0.53241 \n",
      "Epoch: 16, Train Loss: 0.40228, Train Mean IU: 0.52492,                 Valid Loss: 0.41696, Valid Mean IU: 0.53257 \n",
      "Epoch: 17, Train Loss: 0.39653, Train Mean IU: 0.53102,                 Valid Loss: 0.40270, Valid Mean IU: 0.53717 \n",
      "Epoch: 18, Train Loss: 0.38549, Train Mean IU: 0.53907,                 Valid Loss: 0.40644, Valid Mean IU: 0.53137 \n",
      "Epoch: 19, Train Loss: 0.37012, Train Mean IU: 0.54341,                 Valid Loss: 0.39770, Valid Mean IU: 0.53618 \n",
      "Epoch: 20, Train Loss: 0.36531, Train Mean IU: 0.54428,                 Valid Loss: 0.39721, Valid Mean IU: 0.53798 \n",
      "Epoch: 21, Train Loss: 0.35701, Train Mean IU: 0.54932,                 Valid Loss: 0.39131, Valid Mean IU: 0.53669 \n",
      "Epoch: 22, Train Loss: 0.34262, Train Mean IU: 0.55091,                 Valid Loss: 0.38705, Valid Mean IU: 0.53786 \n",
      "Epoch: 23, Train Loss: 0.33300, Train Mean IU: 0.55677,                 Valid Loss: 0.38374, Valid Mean IU: 0.54537 \n",
      "Epoch: 24, Train Loss: 0.33618, Train Mean IU: 0.55542,                 Valid Loss: 0.38625, Valid Mean IU: 0.53881 \n",
      "Epoch: 25, Train Loss: 0.32923, Train Mean IU: 0.55969,                 Valid Loss: 0.39274, Valid Mean IU: 0.52757 \n",
      "Epoch: 26, Train Loss: 0.32419, Train Mean IU: 0.56005,                 Valid Loss: 0.38133, Valid Mean IU: 0.53877 \n",
      "Epoch: 27, Train Loss: 0.31409, Train Mean IU: 0.57064,                 Valid Loss: 0.37719, Valid Mean IU: 0.53900 \n",
      "Epoch: 28, Train Loss: 0.31359, Train Mean IU: 0.56911,                 Valid Loss: 0.37789, Valid Mean IU: 0.53884 \n",
      "Epoch: 29, Train Loss: 0.30378, Train Mean IU: 0.57381,                 Valid Loss: 0.37096, Valid Mean IU: 0.54133 \n",
      "Epoch: 30, Train Loss: 0.30073, Train Mean IU: 0.58053,                 Valid Loss: 0.37705, Valid Mean IU: 0.54011 \n",
      "Epoch: 31, Train Loss: 0.29481, Train Mean IU: 0.58506,                 Valid Loss: 0.37817, Valid Mean IU: 0.53810 \n",
      "Epoch: 32, Train Loss: 0.29196, Train Mean IU: 0.58306,                 Valid Loss: 0.36838, Valid Mean IU: 0.54817 \n",
      "Epoch: 33, Train Loss: 0.28702, Train Mean IU: 0.58325,                 Valid Loss: 0.36649, Valid Mean IU: 0.54689 \n",
      "Epoch: 34, Train Loss: 0.28232, Train Mean IU: 0.59623,                 Valid Loss: 0.37152, Valid Mean IU: 0.54318 \n",
      "Epoch: 35, Train Loss: 0.27740, Train Mean IU: 0.58919,                 Valid Loss: 0.36839, Valid Mean IU: 0.53862 \n",
      "Epoch: 36, Train Loss: 0.27236, Train Mean IU: 0.59353,                 Valid Loss: 0.36212, Valid Mean IU: 0.55020 \n",
      "Epoch: 37, Train Loss: 0.26316, Train Mean IU: 0.60021,                 Valid Loss: 0.36135, Valid Mean IU: 0.54451 \n",
      "Epoch: 38, Train Loss: 0.26362, Train Mean IU: 0.60024,                 Valid Loss: 0.36299, Valid Mean IU: 0.53559 \n",
      "Epoch: 39, Train Loss: 0.26524, Train Mean IU: 0.60670,                 Valid Loss: 0.36152, Valid Mean IU: 0.53983 \n",
      "Epoch: 40, Train Loss: 0.25813, Train Mean IU: 0.60653,                 Valid Loss: 0.36855, Valid Mean IU: 0.53919 \n",
      "Epoch: 41, Train Loss: 0.25044, Train Mean IU: 0.60909,                 Valid Loss: 0.35948, Valid Mean IU: 0.54836 \n",
      "Epoch: 42, Train Loss: 0.24806, Train Mean IU: 0.60937,                 Valid Loss: 0.36096, Valid Mean IU: 0.55124 \n",
      "Epoch: 43, Train Loss: 0.25344, Train Mean IU: 0.61131,                 Valid Loss: 0.36593, Valid Mean IU: 0.54342 \n",
      "Epoch: 44, Train Loss: 0.24650, Train Mean IU: 0.61081,                 Valid Loss: 0.35815, Valid Mean IU: 0.54840 \n",
      "Epoch: 47, Train Loss: 0.23792, Train Mean IU: 0.62197,                 Valid Loss: 0.35821, Valid Mean IU: 0.55112 \n",
      "Epoch: 48, Train Loss: 0.23765, Train Mean IU: 0.62445,                 Valid Loss: 0.36050, Valid Mean IU: 0.54794 \n",
      "Epoch: 49, Train Loss: 0.23098, Train Mean IU: 0.63008,                 Valid Loss: 0.35734, Valid Mean IU: 0.55623 \n",
      "Epoch: 50, Train Loss: 0.23409, Train Mean IU: 0.63316,                 Valid Loss: 0.36620, Valid Mean IU: 0.54893 \n",
      "Epoch: 51, Train Loss: 0.22637, Train Mean IU: 0.63655,                 Valid Loss: 0.35512, Valid Mean IU: 0.55228 \n",
      "Epoch: 52, Train Loss: 0.22158, Train Mean IU: 0.63689,                 Valid Loss: 0.35593, Valid Mean IU: 0.55442 \n",
      "Epoch: 53, Train Loss: 0.22060, Train Mean IU: 0.64307,                 Valid Loss: 0.35869, Valid Mean IU: 0.56010 \n",
      "Epoch: 54, Train Loss: 0.22050, Train Mean IU: 0.64211,                 Valid Loss: 0.35966, Valid Mean IU: 0.54604 \n",
      "Epoch: 55, Train Loss: 0.21882, Train Mean IU: 0.64335,                 Valid Loss: 0.35393, Valid Mean IU: 0.55634 \n",
      "Epoch: 56, Train Loss: 0.21689, Train Mean IU: 0.63954,                 Valid Loss: 0.35273, Valid Mean IU: 0.55231 \n",
      "Epoch: 57, Train Loss: 0.21501, Train Mean IU: 0.64380,                 Valid Loss: 0.35567, Valid Mean IU: 0.55511 \n",
      "Epoch: 58, Train Loss: 0.22048, Train Mean IU: 0.64015,                 Valid Loss: 0.35797, Valid Mean IU: 0.55672 \n",
      "Epoch: 59, Train Loss: 0.21230, Train Mean IU: 0.64918,                 Valid Loss: 0.35751, Valid Mean IU: 0.55350 \n",
      "Epoch: 60, Train Loss: 0.20756, Train Mean IU: 0.65064,                 Valid Loss: 0.35097, Valid Mean IU: 0.55815 \n",
      "Epoch: 61, Train Loss: 0.20700, Train Mean IU: 0.64838,                 Valid Loss: 0.35512, Valid Mean IU: 0.55746 \n",
      "Epoch: 62, Train Loss: 0.20510, Train Mean IU: 0.65138,                 Valid Loss: 0.35398, Valid Mean IU: 0.55860 \n",
      "Epoch: 63, Train Loss: 0.20155, Train Mean IU: 0.65684,                 Valid Loss: 0.35592, Valid Mean IU: 0.55616 \n",
      "Epoch: 64, Train Loss: 0.19938, Train Mean IU: 0.66669,                 Valid Loss: 0.35182, Valid Mean IU: 0.55258 \n",
      "Epoch: 65, Train Loss: 0.20291, Train Mean IU: 0.66149,                 Valid Loss: 0.36057, Valid Mean IU: 0.55194 \n",
      "Epoch: 66, Train Loss: 0.20901, Train Mean IU: 0.65127,                 Valid Loss: 0.35885, Valid Mean IU: 0.54940 \n",
      "Epoch: 67, Train Loss: 0.20011, Train Mean IU: 0.65655,                 Valid Loss: 0.35446, Valid Mean IU: 0.55681 \n",
      "Epoch: 68, Train Loss: 0.19918, Train Mean IU: 0.66490,                 Valid Loss: 0.35808, Valid Mean IU: 0.54631 \n",
      "Epoch: 69, Train Loss: 0.19373, Train Mean IU: 0.67078,                 Valid Loss: 0.35594, Valid Mean IU: 0.55810 \n",
      "Epoch: 70, Train Loss: 0.19337, Train Mean IU: 0.66598,                 Valid Loss: 0.35625, Valid Mean IU: 0.55470 \n",
      "Epoch: 71, Train Loss: 0.19464, Train Mean IU: 0.66977,                 Valid Loss: 0.35734, Valid Mean IU: 0.55656 \n",
      "Epoch: 72, Train Loss: 0.19115, Train Mean IU: 0.66531,                 Valid Loss: 0.35962, Valid Mean IU: 0.55484 \n",
      "Epoch: 73, Train Loss: 0.18904, Train Mean IU: 0.67183,                 Valid Loss: 0.35523, Valid Mean IU: 0.55965 \n",
      "Epoch: 74, Train Loss: 0.18701, Train Mean IU: 0.67706,                 Valid Loss: 0.35559, Valid Mean IU: 0.56270 \n",
      "Epoch: 75, Train Loss: 0.18532, Train Mean IU: 0.67852,                 Valid Loss: 0.35700, Valid Mean IU: 0.55802 \n",
      "Epoch: 76, Train Loss: 0.18355, Train Mean IU: 0.67861,                 Valid Loss: 0.35526, Valid Mean IU: 0.56074 \n",
      "Epoch: 77, Train Loss: 0.18212, Train Mean IU: 0.67792,                 Valid Loss: 0.35891, Valid Mean IU: 0.56053 \n",
      "Epoch: 78, Train Loss: 0.18786, Train Mean IU: 0.67716,                 Valid Loss: 0.35942, Valid Mean IU: 0.55670 \n",
      "Epoch: 79, Train Loss: 0.18157, Train Mean IU: 0.67877,                 Valid Loss: 0.36332, Valid Mean IU: 0.55336 \n",
      "Epoch: 80, Train Loss: 0.17864, Train Mean IU: 0.67912,                 Valid Loss: 0.35934, Valid Mean IU: 0.55784 \n",
      "Epoch: 81, Train Loss: 0.18159, Train Mean IU: 0.67811,                 Valid Loss: 0.36361, Valid Mean IU: 0.55080 \n",
      "Epoch: 82, Train Loss: 0.17674, Train Mean IU: 0.68856,                 Valid Loss: 0.35939, Valid Mean IU: 0.55804 \n",
      "Epoch: 83, Train Loss: 0.17754, Train Mean IU: 0.68163,                 Valid Loss: 0.35650, Valid Mean IU: 0.56003 \n",
      "Epoch: 84, Train Loss: 0.17587, Train Mean IU: 0.68845,                 Valid Loss: 0.35801, Valid Mean IU: 0.56153 \n",
      "Epoch: 85, Train Loss: 0.17307, Train Mean IU: 0.69317,                 Valid Loss: 0.36079, Valid Mean IU: 0.56053 \n",
      "Epoch: 86, Train Loss: 0.17605, Train Mean IU: 0.69264,                 Valid Loss: 0.36227, Valid Mean IU: 0.55722 \n",
      "Epoch: 87, Train Loss: 0.17238, Train Mean IU: 0.69406,                 Valid Loss: 0.35920, Valid Mean IU: 0.56068 \n",
      "Epoch: 88, Train Loss: 0.17555, Train Mean IU: 0.69160,                 Valid Loss: 0.35403, Valid Mean IU: 0.56379 \n",
      "Epoch: 89, Train Loss: 0.16974, Train Mean IU: 0.69333,                 Valid Loss: 0.35748, Valid Mean IU: 0.56099 \n",
      "Epoch: 90, Train Loss: 0.16666, Train Mean IU: 0.69789,                 Valid Loss: 0.35881, Valid Mean IU: 0.56111 \n",
      "Epoch: 91, Train Loss: 0.16917, Train Mean IU: 0.69627,                 Valid Loss: 0.35717, Valid Mean IU: 0.55923 \n",
      "Epoch: 92, Train Loss: 0.16598, Train Mean IU: 0.69949,                 Valid Loss: 0.36216, Valid Mean IU: 0.56072 \n",
      "Epoch: 93, Train Loss: 0.16676, Train Mean IU: 0.69952,                 Valid Loss: 0.36246, Valid Mean IU: 0.56023 \n",
      "Epoch: 94, Train Loss: 0.16626, Train Mean IU: 0.70169,                 Valid Loss: 0.35986, Valid Mean IU: 0.55690 \n",
      "Epoch: 95, Train Loss: 0.16536, Train Mean IU: 0.70418,                 Valid Loss: 0.36022, Valid Mean IU: 0.56081 \n",
      "Epoch: 96, Train Loss: 0.16355, Train Mean IU: 0.69679,                 Valid Loss: 0.35987, Valid Mean IU: 0.56825 \n",
      "Epoch: 97, Train Loss: 0.16510, Train Mean IU: 0.70101,                 Valid Loss: 0.36438, Valid Mean IU: 0.56005 \n",
      "Epoch: 98, Train Loss: 0.16030, Train Mean IU: 0.70442,                 Valid Loss: 0.35873, Valid Mean IU: 0.56475 \n",
      "Epoch: 99, Train Loss: 0.17150, Train Mean IU: 0.69863,                 Valid Loss: 0.36897, Valid Mean IU: 0.54492 \n",
      "Epoch: 100, Train Loss: 0.16071, Train Mean IU: 0.70205,                 Valid Loss: 0.36214, Valid Mean IU: 0.55787 \n",
      "Epoch: 101, Train Loss: 0.15856, Train Mean IU: 0.70905,                 Valid Loss: 0.36701, Valid Mean IU: 0.56023 \n",
      "Epoch: 102, Train Loss: 0.15892, Train Mean IU: 0.71011,                 Valid Loss: 0.36334, Valid Mean IU: 0.55782 \n",
      "Epoch: 103, Train Loss: 0.15862, Train Mean IU: 0.70630,                 Valid Loss: 0.36444, Valid Mean IU: 0.56146 \n",
      "Epoch: 104, Train Loss: 0.15535, Train Mean IU: 0.71606,                 Valid Loss: 0.35748, Valid Mean IU: 0.55870 \n",
      "Epoch: 105, Train Loss: 0.15730, Train Mean IU: 0.71296,                 Valid Loss: 0.35924, Valid Mean IU: 0.55809 \n",
      "Epoch: 106, Train Loss: 0.15835, Train Mean IU: 0.71115,                 Valid Loss: 0.36066, Valid Mean IU: 0.56516 \n",
      "Epoch: 107, Train Loss: 0.16147, Train Mean IU: 0.71304,                 Valid Loss: 0.36290, Valid Mean IU: 0.55453 \n",
      "Epoch: 108, Train Loss: 0.15417, Train Mean IU: 0.71553,                 Valid Loss: 0.36100, Valid Mean IU: 0.55770 \n",
      "Epoch: 109, Train Loss: 0.15615, Train Mean IU: 0.71061,                 Valid Loss: 0.36085, Valid Mean IU: 0.55996 \n",
      "Epoch: 110, Train Loss: 0.15479, Train Mean IU: 0.71703,                 Valid Loss: 0.36041, Valid Mean IU: 0.55547 \n",
      "Epoch: 111, Train Loss: 0.15472, Train Mean IU: 0.71847,                 Valid Loss: 0.35937, Valid Mean IU: 0.56613 \n",
      "Epoch: 112, Train Loss: 0.15152, Train Mean IU: 0.72358,                 Valid Loss: 0.36531, Valid Mean IU: 0.55716 \n",
      "Epoch: 113, Train Loss: 0.15582, Train Mean IU: 0.71423,                 Valid Loss: 0.36615, Valid Mean IU: 0.55969 \n",
      "Epoch: 114, Train Loss: 0.15461, Train Mean IU: 0.72065,                 Valid Loss: 0.36724, Valid Mean IU: 0.55461 \n",
      "Epoch: 115, Train Loss: 0.14995, Train Mean IU: 0.71346,                 Valid Loss: 0.36076, Valid Mean IU: 0.55784 \n",
      "Epoch: 116, Train Loss: 0.14921, Train Mean IU: 0.71660,                 Valid Loss: 0.36236, Valid Mean IU: 0.56401 \n",
      "Epoch: 117, Train Loss: 0.14605, Train Mean IU: 0.72722,                 Valid Loss: 0.36342, Valid Mean IU: 0.56635 \n",
      "Epoch: 118, Train Loss: 0.14499, Train Mean IU: 0.72452,                 Valid Loss: 0.36310, Valid Mean IU: 0.56646 \n",
      "Epoch: 119, Train Loss: 0.15203, Train Mean IU: 0.71192,                 Valid Loss: 0.36384, Valid Mean IU: 0.56060 \n"
     ]
    }
   ],
   "source": [
    "# 由于没有验证集，这里我们每训练完一次模型便在测试集上进行测试\n",
    "\n",
    "# 不显示无关信息\n",
    "np.seterr(divide='ignore',invalid='ignore')\n",
    "\n",
    "\n",
    "Epoch = 120\n",
    "for epoch in range(Epoch):\n",
    "    \n",
    "    # 训练损失\n",
    "    train_loss = 0\n",
    "    # 训练mIoU\n",
    "    train_mean_iu = 0\n",
    "    \n",
    "    # 开始训练\n",
    "    model = model.train()\n",
    "\n",
    "    for data in train_data:\n",
    "        x = data[0].to(device)\n",
    "        y = data[1].to(device)\n",
    "\n",
    "        # 梯度清零\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # 获得模型预测概率分布\n",
    "        outputs = model(x)\n",
    "        # 获得模型预测的标签，大小为(b,n,h,w)\n",
    "        outputs = F.log_softmax(outputs, dim=1)\n",
    "        \n",
    "        # 计算损失\n",
    "        loss = criterion(outputs, y)\n",
    "        \n",
    "        # 反向传播\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        # 记录loss\n",
    "        train_loss += loss.item()\n",
    "        \n",
    "        # 获得预测标签\n",
    "        label_pred = outputs.max(dim=1)[1].data.cpu().numpy()\n",
    "        label_true = y.data.cpu().numpy()\n",
    "        \n",
    "        for lbt, lbp in zip(label_true, label_pred):\n",
    "            # 返回每张图的pred和gt标签计算mIoU\n",
    "            mean_iu = mIoU(lbt, lbp, num_classes)\n",
    "            train_mean_iu += mean_iu\n",
    "    \n",
    "\n",
    "    # 测试损失\n",
    "    eval_loss = 0\n",
    "    # 测试mIoU\n",
    "    eval_mean_iu = 0\n",
    "    \n",
    "    # 进行模型测试\n",
    "    model = model.eval()\n",
    "    \n",
    "    for data in test_data:\n",
    "        x_test = data[0].to(device)\n",
    "        y_test = data[1].to(device)\n",
    "        outputs_test = model(x_test)\n",
    "        outputs_test = F.log_softmax(outputs_test, dim=1)\n",
    "        \n",
    "        loss = criterion(outputs_test, y_test)\n",
    "        eval_loss += loss.item()\n",
    "        \n",
    "        label_pred = outputs_test.max(dim=1)[1].data.cpu().numpy()\n",
    "        label_true = y_test.data.cpu().numpy()\n",
    "        \n",
    "        for lbt, lbp in zip(label_true, label_pred):\n",
    "            mean_iu = mIoU(lbt, lbp, num_classes)\n",
    "            eval_mean_iu += mean_iu\n",
    "        \n",
    "\n",
    "    epoch_str = ('Epoch: {}, Train Loss: {:.5f}, Train Mean IU: {:.5f},\\\n",
    "                 Valid Loss: {:.5f}, Valid Mean IU: {:.5f} '.format(\n",
    "        epoch, train_loss / len(train_data), train_mean_iu / len(voc_train), \n",
    "        eval_loss / len(test_data), eval_mean_iu / len(voc_test)))\n",
    "   \n",
    "    print(epoch_str)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们随机选择一些图像，并展示其真实标签与模型的预测标签。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 4159548 characters]",
      "text/plain": [
       "<Figure size 6000x5000 with 30 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 将预测的标签映射到colormap上\n",
    "cm = np.array(colormap).astype('uint8')\n",
    "\n",
    "def predict(im, label):\n",
    "    im = Variable(im.unsqueeze(0)).cuda()\n",
    "    out = model(im)\n",
    "    pred = out.max(1)[1].squeeze().cpu().data.numpy()\n",
    "    pred = cm[pred]\n",
    "    return pred, cm[label.numpy()]\n",
    "\n",
    "\n",
    "_, figs = plt.subplots(10, 3, figsize=(12, 10))\n",
    "\n",
    "for i in range(10):\n",
    "    x, y = voc_test[i]\n",
    "    x.to(device)\n",
    "    y.to(device)\n",
    "    pred, label = predict(x, y)\n",
    "    figs[i, 0].imshow(Image.open(voc_test.data_list[i]))\n",
    "    figs[i, 0].axes.get_xaxis().set_visible(False)\n",
    "    figs[i, 0].axes.get_yaxis().set_visible(False)\n",
    "    figs[i, 1].imshow(label)\n",
    "    figs[i, 1].axes.get_xaxis().set_visible(False)\n",
    "    figs[i, 1].axes.get_yaxis().set_visible(False)\n",
    "    figs[i, 2].imshow(pred)\n",
    "    figs[i, 2].axes.get_xaxis().set_visible(False)\n",
    "    figs[i, 2].axes.get_yaxis().set_visible(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.5 小结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在本章中，我们学习了语义分割的相关知识。我们了解了分割网络与分类网络的区别，在此基础上学习了用于语义分割的网络FCN，之后动手实现了FCN的相应功能并在Pascal Voc数据集上进行了训练和测试。作为计算机视觉的重要研究内容之一，语义分割为后续的图像语义理解如目标检测、实例分割等奠定了一定的基础。接下来，我们将继续学习另一种图像语义理解的研究——目标检测。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.6 习题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 习题11.1：FCN中，融合主干网络不同层次输出的特征图是通过跳跃连接后逐像素相加，你还能想到别的融合方式吗？请动手实现你的融合方式，并和原来的融合方式进行比较。\n",
    "\n",
    "#### 习题11.2：如果不同类别的像素个数差别很大，那么采用本章中介绍的逐像素交叉熵损失函数训练FCN做语义分割可能存在什么问题？如果存在问题，应该如何修改逐像素交叉熵损失函数？\n",
    "\n",
    "#### 习题11.3：在简介（章节11.1）中，我们说语义分割需要建模条件概率$p(\\mathbf{Y}|\\mathbf{I})$，但是在FCN中我们其实建模的是每个像素的条件概率$p(y_i|\\mathbf{I})$。当然这是因为$\\mathbf{Y}$的求解空间太大，所采取的一种近似求解。那么这种近似会导致FCN的分割结果出现什么问题？\n",
    "\n",
    "#### 习题11.4：在第9章中，我们介绍了无监督图像分割。无监督图像分割中的思想或者方法是否可以帮助改进FCN，提升语义分割的结果？请给出你的解决思路，并动手实现予以验证。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 参考文献"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, Andrew Zisserman: The Pascal Visual Object Classes (VOC) Challenge. Int. J. Comput. Vis. 88(2): 303-338 (2010)\n",
    "\n",
    "[2] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele: The Cityscapes Dataset for Semantic Urban Scene Understanding. CVPR 2016: 3213-3223\n",
    "\n",
    "[3] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, Antonio Torralba: Semantic Understanding of Scenes through the ADE20K Dataset. CoRR abs/1608.05442 (2016)\n",
    "\n",
    "[4] Jonathan Long, Evan Shelhamer, Trevor Darrell: Fully convolutional networks for semantic segmentation. CVPR 2015: 3431-3440\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
