{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第14章 人体姿态估计"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.1 简介"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "人体姿态估计（human pose estimation）是计算机视觉的一个重要任务，其目的是识别人体的动作和姿势。这对于很多应用来说都非常重要，例如运动分析、虚拟现实和人机交互等。在虚拟现实中，人体姿态识别技术可以帮助跟踪用户的动作，使得虚拟现实体验更加逼真；在人机交互中，人体姿态识别技术可以帮助设备更好地识别用户的命令，从而使得人机交互更加流畅。\n",
    "\n",
    "一般来说，人体姿态估计任务分为二维姿态估计和三维姿态估计，它们之间有一些明显的差异。\n",
    "二维姿态估计是指检测人体在图像中的关节点坐标，输出为二维坐标。所以二维姿态估计类似于语义分割和目标检测这种视觉识别任务，所需技术包括图像特征提取和坐标回归等。\n",
    "三维姿态估计是指识别人体在三维空间中的关节点坐标，输出为三维坐标。所以三维姿态估计通常还需要场景重建技术，如立体视觉、运动估计等。\n",
    "\n",
    "在本章中，我们主要介绍二维姿态估计，用数学语言描述该任务即是：预定义$N$个关节点，给定$n$张图像，定义图像空间$\\mathbb{I}$，给定数据集$\\mathcal{D} = \\{(\\boldsymbol{I}^{(n)},\\boldsymbol{y}^{(n)})\\}_{n=1}^N$，其中$\\boldsymbol{I}^{(n)}\\in \\mathbb{I}$是数据集中的第$n$张图像，$\\boldsymbol{y}^{(n)}=(\\boldsymbol{y}^{(n)}_j;j=1 \\ldots N)\\in\\mathbb{R}^{N \\times 3}$是这张图像中真实的人体关节点坐标向量，$\\boldsymbol{y}^{(n)}_j$是第$j$个关节点的坐标。姿态估计的任务是从$\\mathcal{D}$中学习得到一个从图像空间到人体关节点坐标向量空间的映射$f:\\mathbb{I} \\rightarrow \\mathbb{R}^{N \\times 3}$，从而给定任意一张测试图像$\\boldsymbol{I}$，可以用学习得到的映射函数$f$预测其标签：$\\hat{\\boldsymbol{y}}=f(\\boldsymbol{I})$。由于根据人体结构，可以把关节点连接起来，形成人体骨架（skeleton）的形式（如图14-1所示），所以人体姿势估计也叫做人体骨架估计。\n",
    "接下来，我们将探讨一些经典的人体姿态估计方法。这些方法已被证明是后续工作的基础，它们的思想在当今仍具有参考价值。\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic4.zhimg.com/80/v2-f535a900c803474f2f0d5c973cfdfdb7_720w.webp\" width=400>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图14-1 人体姿态估计示意图</div>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2 数据集和度量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本节将介绍人体姿态估计相关的数据集和度量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14.2.1 数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在人体姿态估计中，通常会选择以下几种常用的数据集进行模型的训练与测试：\n",
    "1. Leeds Sports Pose (LSP) Dataset $^{[1]}$: 共收录2000张全身姿势图像，这些图像被分为8类（田径、羽毛球、棒球、体操、跑酷、足球、网球和排球），每个人体图像都有14个关节点。\n",
    "1. Frames Labeled In Cinema (FLIC) Dataset $^{[2]}$: 共收录5003幅图像，这些图像来自30部著名好莱坞电影，每部电影每10帧进行一次人体检测，约20000个人体候选区被发布到众包平台Amazon Mechanical Turk并获得10个上半身关节的真实坐标。最后去除严重遮挡或非正视图的图像。\n",
    "1. Max Planck Institute for Informatics Human Pose （MPII） Dataset $^{[3]}$: 该数据集包含超过40000张人物图像，涵盖了491种不同的活动。数据集的图像来源于YouTube，涵盖了室内外场景、各种成像条件，因此该数据集在外观变异性方面具有很高的多样性。研究人员对于收集到的图像提供了丰富的注释。注释内容包括身体关节的位置、头部和躯干的3D视角，以及眼睛和鼻子的位置。此外还对所有身体关节和部位的可见性进行了注释。\n",
    "4. Microsoft Common Objects in Context（MS COCO）Dataset $^{[4]}$：MS COCO数据集不仅可以用于⽬标检测、分割、关键点检测等任务，也可以用于人体姿态估计。MS COCO对每个人体实例都标注了17个关键点，包括鼻子、两眼、两耳、肩部、手肘、手腕、髋部、膝盖和脚踝。相比于MPII数据集，MS COCO在人脸结构的标注上更为丰富和多样化。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14.2.2 评测指标"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在人体姿态估计中，常常用以下评测指标：\n",
    "1. Percentage of Correct Parts (PCP)：检测到正确部位的比例。\n",
    "PCP一般用于衡量肢（limb）是否预测正确，判断标准是：如果预测关节位置与真实关节位置之间的距离小于当前肢体长度的一半，那么就认为这根骨架预测正确。\n",
    "<!-- 在PCP中认为一个骨架段（limb）被检测到的标准是：骨架段的两个关节点的位置与真实关节点的位置的距离小于骨架段的长度的一半（因此常被写作PCP@0.5）。\n",
    "    $$\n",
    "PCP@0.5 = \\frac{1}{K} \\sum_{i=1}^{K} \\frac{\\sum_{j=1}^{M} \\delta(d_{i,j} < 0.5 L_{i,j})}{M}\n",
    "    $$ -->\n",
    "2. Percentage of Correct Keypoints (PCK)：关节点被准确检测的比例。\n",
    "计算检测的关节点与其对应的真值间的归一化距离小于设定阈值的比例。\n",
    "3. Object Keypoint Similarity (OKS)：关节点相似度。\n",
    "具体计算公式为：\n",
    "    $$\n",
    "    OKS = \\frac{\\sum_{i} \\exp \\left(-d_{i}^{2} / 2 s^{2} k_{i}^{2}\\right) \\delta\\left(v_{i}>0\\right)}{\\delta\\left(v_{i}>0\\right)}\n",
    "    $$\n",
    "    其中$d_i$是检测到的关节点与相应的真值之间的欧几里得距离，$v_i$是真值的可见性标志，$v_{{i}}=1$表示这个关节点的可见性为1，即关节点无遮挡并且已标注，$v_{i}=2$表示关节点有遮挡但已标注。$s$是尺度因子，其值为人体检测框面积的平方根$\\sqrt{H \\times W}$，这里的$H, W$是行人检测框的高和宽。\n",
    "    $k_i$表示该关节点的归一化因子，其是通过对已有数据集中所有的真值计算得到的标准差而获得，反映了当前关节点对整体影响程度。值越大，说明在整个数据集中对该点的标注效果越差；值越小，说明整个数据集中对该点的标注效果越好。\n",
    "    可以看出，OKS对于关节点的可见性，尺度因子还有关节点标注效果都进行了考虑，因此能够更加综合地评估人体姿态估计的性能。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3 算法简述"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了更加精确地预测人体的姿态，人们从不同方面进行了大量的探索。\n",
    "1. 整体视图。例如，关节位置回归、热度图 $^{[6]}$回归。\n",
    "2. 长距离空间关系。例如，卷积姿态机（convolutional pose machines, CMPs）$^{[7]}$。\n",
    "3. 迭代估计。例如，迭代误差反馈（iterative error feedback）$^{[8]}$。\n",
    "4. 网络架构。例如，堆叠沙漏网络（stacked hourglass networks）$^{[9]}$。\n",
    "\n",
    "接下来我们将介绍一些基于深度学习的人体姿态估计的方法，体会他们在设计时的思考与各自的局限性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14.3.1 DeepPose\n",
    "正如在简介中介绍的，人体姿态估计的目的是获得映射函数$f$。对于给定的任意一张测试图像$\\boldsymbol{I}$，可以用学习得到的映射函数$f$预测其标签：$\\hat{\\boldsymbol{y}}=f(\\boldsymbol{I})$。让我们回忆一下第12章目标检测中，我们详细学习了候选框回归，在这里也用到同样的思想，对关键点的坐标进行回归，就可以估计人体的姿态了。\n",
    "\n",
    "2014年Alexander Toshev和Christian Szegedy提出的DeepPose $^{[5]}$ 就是秉承着这个想法，这也是首个基于深度学习的人体姿态估计算法。这篇文章构建了一个级联的回归网络，使用深度神经网络来实现人体姿态估计，并将人体姿态估计看作是一个人体关节的回归问题。\n",
    "\n",
    "如图 14-2 所示，图像的局部信息很难提供有效的姿态信息，作者希望能够采用一种更加全面的方法，构建出一种更通用、更具端到端性质的设计，不需要涉及人体学方面的知识，只要将大量的样例输入，便可获得所需的模型。而正好那几年，计算机视觉领域出现了卷积神经网络这一有潜力的工具，因此便有了DeepPose的出现。\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic2.zhimg.com/80/v2-5430909cbd85831f1e32c88ff90b9d75_1440w.webp\" width=300>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图14-2 单靠局部信息很难有效预测姿态（插图来自参考文献[3]）</div>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "参考文献[5]认为，这种方法有两个优点：（1）深度神经网络具有捕获全局上下文信息的能力，每一个关节的回归器都用到整张图像的信息；（2）比起以往的图模型方法，这种表示方法更加简单，不需要设计人体部分的特征表示或者检测器，也不需要设计人体关节之间的拓扑模型和交互。\n",
    "<!--   -->\n",
    "整个DeepPose架构分为初始阶段（initial stage）和后续的精化阶段（refinement stage）。如图 14-3 所示，DeepPose首先将整张图像作为输入，使用卷积神经网络进行全局特征提取，从而初步估计出人体关节点。在这一阶段主要是最小化预测和真实姿态向量之间的$L_2$距离。\n",
    "$$\n",
    "\\arg \\min _{\\theta} \\sum_{(\\boldsymbol{I}, y) \\in {D_{N}}}~\\sum_{i = 1}^{k}\\left\\|\\boldsymbol{y}_{i}^{(n)}-\\psi_{i}(\\boldsymbol{I}^{(n)} ; \\theta)\\right\\|_{2}^{2}\n",
    "$$\n",
    "上式中，$k$是需要预测的肢体总数，$\\theta$是网络模型的参数，$\\boldsymbol{y}_{i}^{(n)}$是第$i$个节点的真值姿态向量，$\\psi_{i}(\\boldsymbol{I}^{(n)} ; \\theta)$是网络预测的姿态向量。\n",
    "\n",
    "<div style=\"text-align:center;\">\n",
    "    <!-- 图像 A -->\n",
    "    <div style=\"display: inline-block; margin-bottom: 10px;\">\n",
    "        <img style=\"border-radius: 0.3125em;\" \n",
    "        src=\"https://pic1.zhimg.com/v2-6361b51a8d77490befc15273a912092c_r.jpg\" width=600>\n",
    "        <div style=\"color:orange; \n",
    "        display: block;\n",
    "        color: #999;\n",
    "        padding: 2px;\">(a) 初始阶段示意图</div>\n",
    "    </div>\n",
    "    <br>\n",
    "    <!-- 图像 B -->\n",
    "    <!-- (您可以在这里插入另一个图像元素，就像上面的示例一样) -->\n",
    "    <div style=\"display: inline-block; margin-top: 10px;\">\n",
    "        <img style=\"border-radius: 0.3125em;\" \n",
    "        src=\"https://pic4.zhimg.com/80/v2-36e5318a6b3470b2a859eb6832fa6e87_720w.webp\" width=600>\n",
    "        <div style=\"color:orange; \n",
    "        display: block;\n",
    "        color: #999;\n",
    "        padding: 2px;\">(b) 精化阶段示意图</div>\n",
    "    </div>\n",
    "    图14-3 DeepPose流程（插图来自参考文献[5]）\n",
    "</div>\n",
    "\n",
    "由于初始阶段网络的输入尺寸固定为220 $\\times$ 220，网络难以兼顾细节信息和全局信息，因此作者设计了精化阶段。在这一阶段，作者通过多阶段优化前一阶段的预测结果，也就是初步的节点坐标，然后在原始图像中根据该坐标选择一定的局部区域作为候选框，达到扩充数据，增加细节的效果，从而提高了回归精度。\n",
    "\n",
    "总的来说，DeepPose相对于以往的方法有如下创新点：\n",
    "1. 对于如何使用全局观点获得关节点的坐标的问题，作者提出了使用卷积神经网络作为全局特征提取，将整张图像作为输入的方法。\n",
    "2. 人体关节点本身存在分布规律，而关节点的坐标在图像坐标系下并无特别的规律，因此作者提出了在每一个环节中，通过中点、长和宽选择一个包围盒，然后计算节点在包围盒中的坐标，从而将绝对坐标转换为统一的坐标。\n",
    "3. 为了实现更高精度的坐标计算，作者提出了级联的卷积神经网络，在最初输入时，将图像设定为固定大小，通过缩放的方式将图像缩放到这个大小，在获得初步的节点坐标之后，再在原始图像中根据该坐标选择一定的局部区域，从而实现更高准确度的计算节点的坐标。\n",
    "\n",
    "通过这一系列的设计，DeepPose成功地成为了当时的最高水平（state-of-the-art, SOTA），并引领了深度学习在人体姿态估计领域的潮流。接下来，我们来动手编写DeepPose并在LSP数据集上测试预测效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import random\n",
    "import scipy.io\n",
    "from cv2 import copyMakeBorder, flip, BORDER_CONSTANT\n",
    "from tensorflow.keras.utils import multi_gpu_model \n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Conv2D, Flatten, MaxPooling2D, \n",
    "                                Dense, BatchNormalization, Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "coords_file_name = \"lsp_dataset/joints.mat\"\n",
    "joints = scipy.io.loadmat(coords_file_name)\n",
    "joints = joints['joints'][:2,:]\n",
    "W = 220\n",
    "H = 220\n",
    "image_list = []\n",
    "num_image = len(joints[0][0])\n",
    "\n",
    "for i in range(1, num_image+1):\n",
    "    if i<10:\n",
    "        img_name = \"im000{0}.jpg\".format(i)\n",
    "    elif i<100:\n",
    "        img_name = \"im00{0}.jpg\".format(i)\n",
    "    elif i<1000:\n",
    "        img_name = \"im0{0}.jpg\".format(i)\n",
    "    else:\n",
    "        img_name = \"im{0}.jpg\".format(i)\n",
    "\n",
    "    img = plt.imread(\"lsp_dataset/images/{0}\".format(img_name))\n",
    "\n",
    "    if img.shape[0] < H:\n",
    "        height = H - img.shape[0]\n",
    "    else:\n",
    "        height = 0\n",
    "\n",
    "    if img.shape[1] < W:\n",
    "        width = W - img.shape[1]\n",
    "    else:\n",
    "        width = 0\n",
    "\n",
    "    if np.mod(width,2) == 0:\n",
    "        extra_width = 0\n",
    "    else:\n",
    "        extra_width = 1\n",
    "\n",
    "    if np.mod(height,2) == 0:\n",
    "        extra_height = 0\n",
    "    else:\n",
    "        extra_height = 1\n",
    "\n",
    "    joints[0,:,i-1] = (joints[0,:,i-1] + width/2)/W\n",
    "    joints[1,:,i-1] = (joints[1,:,i-1] + height/2)/H\n",
    "    img = copyMakeBorder(img,int(height/2)+extra_height,\n",
    "                         int(height/2),int(width/2)+extra_width,int(width/2),\n",
    "                         BORDER_CONSTANT,value=[0,0,0])\n",
    "    image_list.append(img/255)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_train = 1600\n",
    "num_validation = num_image-num_train\n",
    "W = 220\n",
    "H = 220\n",
    "\n",
    "X_train = np.array(image_list[0:num_train])\n",
    "Y_train = np.array(joints[:,:,0:num_train])\n",
    "\n",
    "X_train = np.reshape(X_train,(num_train,W,H,3))\n",
    "Y_train = np.transpose(np.reshape(Y_train,(28, num_train)))\n",
    "\n",
    "\n",
    "X_validation = np.array(image_list[num_train:])\n",
    "Y_validation = np.array(joints[:,:,num_train:])\n",
    "\n",
    "X_validation = np.reshape(X_validation,(num_validation,W,H,3))\n",
    "Y_validation = np.transpose(np.reshape(Y_validation,\n",
    "                                       (28,num_validation)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(400, 220, 220, 3)\n",
      "(400, 28)\n",
      "(1600, 220, 220, 3)\n",
      "(1600, 28)\n"
     ]
    }
   ],
   "source": [
    "print(X_validation.shape)\n",
    "print(Y_validation.shape)\n",
    "print(X_train.shape)\n",
    "print(Y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LSP joint\n",
    "# 1   Right ankle\n",
    "# 2   Right knee\n",
    "# 3   Right hip\n",
    "# 4   Left hip\n",
    "# 5   Left knee\n",
    "# 6   Left ankle\n",
    "# 7   Right wrist\n",
    "# 8   Right elbow\n",
    "# 9   Right shoulder\n",
    "# 10  Left shoulder\n",
    "# 11  Left elbow\n",
    "# 12  Left wrist\n",
    "# 13  Neck\n",
    "# 14  Head top\n",
    "\n",
    "def PCP_RightAnkle(y_true, y_pred):\n",
    "    joint = 0 \n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "def PCP_RightKnee(y_true, y_pred):\n",
    "    joint = 1\n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "def PCP_RightHip(y_true, y_pred):\n",
    "    joint = 2\n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "def PCP_LeftHip(y_true, y_pred):\n",
    "    joint = 3\n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "def PCP_LeftKnee(y_true, y_pred):\n",
    "    joint = 4\n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "def PCP_LeftAnkle(y_true, y_pred):\n",
    "    joint = 5\n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "def PCP_RightWrist(y_true, y_pred):\n",
    "    joint = 6\n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "def PCP_RightElbow(y_true, y_pred):\n",
    "    joint = 7\n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "def PCP_RightShoulder(y_true, y_pred):\n",
    "    joint = 8\n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "def PCP_LeftShoulder(y_true, y_pred):\n",
    "    joint = 9\n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "def PCP_LeftElbow(y_true, y_pred):\n",
    "    joint = 10\n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "def PCP_LeftWrist(y_true, y_pred):\n",
    "    joint = 11\n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "def PCP_Neck(y_true, y_pred):\n",
    "    joint = 12\n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "def PCP_HeadTop(y_true, y_pred):\n",
    "    joint = 13\n",
    "    x_diff =  y_true[joint] - y_pred[joint]\n",
    "    y_diff =  y_true[joint+14] - y_pred[joint+14]\n",
    "    diff = tf.keras.backend.sqrt(tf.math.pow(x_diff,\n",
    "                             2) + tf.math.pow(y_diff,2))\n",
    "    return tf.keras.backend.mean(diff<5/W)\n",
    "\n",
    "\n",
    "def l2_loss(y_true, y_pred):\n",
    "    x_diff =  y_true[0:14] - y_pred[0:14]\n",
    "    y_diff =  y_true[14:28] - y_pred[14:28]\n",
    "    diff = tf.math.pow(x_diff,2) + tf.math.pow(y_diff,2)\n",
    "    return tf.keras.backend.sum(diff)/batch_size\n",
    "\n",
    "\n",
    "def lossAccPlot(history):\n",
    "    plt.figure(figsize=(20,10))\n",
    "\n",
    "    plt.figure()\n",
    "    plt.title('Loss')\n",
    "    plt.plot(history.history['loss'], color='b', label='train')\n",
    "    plt.plot(history.history['val_loss'], color='r', \n",
    "             label='validation')\n",
    "    plt.legend()\n",
    "\n",
    "    for i in [\"PCP_RightAnkle\", \"PCP_RightKnee\", \"PCP_RightHip\", \n",
    "              \"PCP_LeftHip\", \"PCP_LeftKnee\", \"PCP_LeftAnkle\", \n",
    "              \"PCP_RightWrist\", \"PCP_RightElbow\", \"PCP_RightShoulder\", \n",
    "              \"PCP_LeftShoulder\", \"PCP_LeftElbow\", \"PCP_LeftWrist\", \n",
    "              \"PCP_Neck\", \"PCP_HeadTop\"]:\n",
    "\n",
    "        plt.figure()\n",
    "        plt.title(i)\n",
    "        plt.plot(history.history[i], color='blue', label='train')\n",
    "        plt.plot(history.history[\"val_\"+i], color='r',\n",
    "                 label='validation')\n",
    "        plt.legend()\n",
    "        plt.ylim([0,1])\n",
    "     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.status.busy": "2023-04-10T15:19:50.839168Z",
     "iopub.status.idle": "2023-04-10T15:19:50.840015Z",
     "shell.execute_reply": "2023-04-10T15:19:50.839720Z",
     "shell.execute_reply.started": "2023-04-10T15:19:50.839686Z"
    }
   },
   "outputs": [],
   "source": [
    "batch_size = 400\n",
    "epochs_number = 120\n",
    "learning_rate = 5e-5\n",
    "momentum = 0.9\n",
    "\n",
    "\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Conv2D(48, (11, 11), activation='relu', strides=4, \n",
    "                 kernel_initializer='he_uniform', padding='same',  \n",
    "                 input_shape=(W, H, 3)))\n",
    "model.add(BatchNormalization())\n",
    "model.add(MaxPooling2D((2, 2)))\n",
    "\n",
    "model.add(Conv2D(128, (5, 5), activation='relu', strides=2, \n",
    "                 kernel_initializer='he_uniform', padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(MaxPooling2D((2, 2)))\n",
    "\n",
    "model.add(Conv2D(192, (3, 3), activation='relu', strides=1, \n",
    "                 kernel_initializer='he_uniform', padding='same'))\n",
    "\n",
    "model.add(Conv2D(192, (3, 3), activation='relu', strides=1, \n",
    "                 kernel_initializer='he_uniform', padding='same'))\n",
    "\n",
    "model.add(Conv2D(192, (3, 3), activation='relu', strides=1, \n",
    "                 kernel_initializer='he_uniform', padding='same'))\n",
    "model.add(BatchNormalization())\n",
    "model.add(MaxPooling2D((2, 2), padding='same'))\n",
    "\n",
    "model.add(Flatten())\n",
    "\n",
    "model.add(Dense(4096, activation='relu'))\n",
    "model.add(Dropout(0.4))\n",
    "model.add(Dense(4096, activation='relu'))\n",
    "model.add(Dropout(0.4))\n",
    "model.add(Dense(28, activation='linear'))\n",
    "\n",
    "#model = multi_gpu_model(model, gpus=4)\n",
    "\n",
    "\n",
    "optimizer = tf.keras.optimizers.Adam(0.0005)\n",
    "model.compile(loss=l2_loss , optimizer=optimizer, \n",
    "              metrics=[PCP_RightAnkle, PCP_RightKnee, PCP_RightHip, \n",
    "                       PCP_LeftHip, PCP_LeftKnee, PCP_LeftAnkle, \n",
    "                       PCP_RightWrist, PCP_RightElbow, PCP_RightShoulder, \n",
    "                       PCP_LeftShoulder, PCP_LeftElbow, PCP_LeftWrist, \n",
    "                       PCP_Neck, PCP_HeadTop])\n",
    "\n",
    "history = model.fit(X_train, Y_train, epochs=epochs_number, \n",
    "                                        batch_size=batch_size, \n",
    "                    validation_data=(X_validation, Y_validation))\n",
    "\n",
    "     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里为了方便，我们不展示具体训练过程的输出。接着我们来观测模型训练的相关图像。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 2000x1000 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 19204 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 28828 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 28024 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 27432 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 26528 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 27784 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 28008 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 28904 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 28816 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 29364 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 28484 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 28936 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 28160 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 27512 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 28576 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lossAccPlot(history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 184496 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 168916 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 92200 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 156624 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 121720 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 144240 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 238292 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 125724 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 146936 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 90664 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predicted = model.predict(X_validation)\n",
    "for img_number in range(10):\n",
    "    plt.figure()\n",
    "    plt.imshow(X_validation[img_number])\n",
    "    plt.scatter(predicted[img_number,0:14]*W, \n",
    "                predicted[img_number,14:28]*W,label='Predicted')\n",
    "    plt.scatter(Y_validation[img_number,0:14]*W, \n",
    "                Y_validation[img_number,14:28]*W,label='True')\n",
    "    plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但是DeepPose的设计仍然存在不少缺陷：\n",
    "* 在复杂区域精度较低；\n",
    "* 将人体姿态估计问题作为一个精确回归问题，没有考虑人体姿态的不确定性；\n",
    "* 如果最初的估计与实际情况相差甚远，那么后续的迭代无法纠正初始的错误。\n",
    "\n",
    "针对以上这些问题，后续的工作进行了大量的改进，14.3.2节将介绍*Stacked Hourglass Networks for Human Pose Estimation*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14.3.2 堆叠沙漏网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alejandro Newell等[9]提出使用堆叠沙漏网络来估计人体姿态，该模型引发了业界的关注并受到了广泛的认可，许多后续的人体姿态估计方法都借鉴了这一网络结构。\n",
    "\n",
    "为了捕捉不同尺度下图像所包含的信息，作者设计了一个复杂的堆叠沙漏模块。局部信息对于特征的提取是必要的，而最终的姿态估计则需要对整体人体的一致理解。从不同尺度下可以提取出许多有用的信息，例如人体的方位、肢体的动作、相邻关节点的关系等。\n",
    "\n",
    "首先让我们来介绍什么是堆叠沙漏结构：堆叠沙漏结构，顾名思义，就是多个沙漏结构堆叠而成的神经网络。那么什么是沙漏结构呢？\n",
    "如图14-4 所示，这种堆叠在一起的沙漏模块结构是对称的，每个沙漏结构都分为自下而上（bottom-up）过程和自上而下（top-down）过程。自下而上过程将图像从高分辨率降到低分辨率，自上而下过程将图像从低分辨率升到高分辨率。这种网络结构包含了许多池化和上采样的步骤，池化用于提取特征减少冗余，上采样可以结合多个分辨率的特征。\n",
    "自下而上先检测图像中人体部件，然后将图像中多人人体的部件分别分配到不同的人体实例上，因此如果只使用自下而上结构，那么在测试推断的时候往往更快速，模型尺寸更小，但因为没能更细致的对人体关节点建模，所以往往准确度稍低。\n",
    "自上而下过程将人体检测和关节点检测分离，在图像上首先运行一个人体检测器，找到所有的人体实例，对每个人体子图再使用关节点检测。但是只使用自上而下往往比较慢，但姿态估计准确度较高。\n",
    "将自下而上过程和自上而下过程结合起来，就能将全局信息与局部信息综合起来考虑回归姿态热度图。\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic2.zhimg.com/80/v2-0e575daf9ba44ca701a37d3c77a5642d_720w.webp\" width=800>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图14-4 网络由多个堆叠沙漏模块组成（插图来自参考文献[9]）</div>\n",
    "</center>\n",
    "\n",
    "图14-5 具体展示了沙漏模块的结构。每个沙漏模块的输入均为64 $\\times$ 64，其结构具有对称性。该模块中采用卷积和池化层来降低特征的分辨率，在每一个池化步骤中，网络将产生分支，并在原来提前池化的分辨率下使用更多的卷积。当达到最低的分辨率时，网络将采用最邻近插值的方式进行上采样，以结合不同尺度下的特征，并将两个特征集按元素位置相加。当达到输出分辨率时，网络会接入两个1 $\\times$ 1的卷积层来进行最后的预测，输出结果为一组热图，其中热图中每个像素点处的概率值代表了该像素点上存在关节的可能性。\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic3.zhimg.com/80/v2-d687f947c1e35ea4508e2ef74e7202fe_720w.webp\" width=600>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图14-5 沙漏模块示意图（插图来自参考文献[9]）</div>\n",
    "</center>\n",
    "\n",
    "图14-6 展示了沙漏模块的基本结构，残差模块，这是一种旁路相加结构。残差模块由两个参数控制：输入深度和输出深度。可以对任意尺寸图像操作。通过残差模块提取了较高层次的特征（卷积路），同时保留了原有层次的信息（跳级路）。\n",
    "\n",
    "<center>\n",
    "    <img style=\"border-radius: 0.3125em;\" \n",
    "    src=\"https://pic1.zhimg.com/80/v2-03aaffb8b495a99f5e93b6f8a407cb30_720w.jpg\" width=250>\n",
    "    <br>\n",
    "    <div style=\"color:orange; \n",
    "    display: inline-block;\n",
    "    color: #999;\n",
    "    padding: 2px;\">图14-6 残差模块</div>\n",
    "</center>\n",
    "网络结构采用了堆叠沙漏模块的形式，从而实现了自底向上和自顶向下的重复过程。作者提出的关键技术是采用中继监督（intermediate supervision）来预测每一个沙漏模块的输出结果，即计算每一个沙漏模块输出的热图损失。需要指出的是，每一个模块的权重是独立的，所有模块都基于相同的真值添加损失函数。\n",
    "\n",
    "我们先编写沙漏模块的代码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.status.busy": "2023-04-10T15:19:50.849176Z",
     "iopub.status.idle": "2023-04-10T15:19:50.850446Z",
     "shell.execute_reply": "2023-04-10T15:19:50.850146Z",
     "shell.execute_reply.started": "2023-04-10T15:19:50.850105Z"
    }
   },
   "outputs": [],
   "source": [
    "# 堆叠沙漏模块\n",
    "class Hourglass(nn.Module):\n",
    "    def __init__(self, n, channel, increase=0):\n",
    "        super(Hourglass, self).__init__()\n",
    "        # 先池化降分辨率\n",
    "        self.pool = Pool(2, 2)\n",
    "        \n",
    "        # 用residual_module来下采样\n",
    "        self.conv = Residual(channel, channel)\n",
    "        self.n = n\n",
    "        \n",
    "        # 使用递归的方式完成u型结构，每次向内递归一层，\n",
    "        # 直到n=1不能再套u型层，此时就通过卷积层\n",
    "        if self.n > 1:\n",
    "            self.smaller_hourglass = Hourglass(n-1, channel)\n",
    "        else:\n",
    "            self.smaller_hourglass = Residual(channel, channel)\n",
    "            \n",
    "        # 最近邻上采样，恢复分辨率，对应于self.pool的池化层\n",
    "        self.up2 = nn.Upsample(scale_factor=2, mode='nearest')\n",
    "\n",
    "    def forward(self, x):\n",
    "        \n",
    "        # 先池化降低分辨率，完成下采样\n",
    "        pool = self.pool(x)\n",
    "        # 用residual_module来Downsample\n",
    "        conv  = self.conv(pool)\n",
    "        #hourglass,不过参数n=n-1\n",
    "        low = self.smaller_hourglass(conv)\n",
    "        # 最近邻上采样\n",
    "        up  = self.up(low)\n",
    "        # 将原图像和操作后图像相加，完成u型结构\n",
    "        return x + up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "堆叠沙漏网络是基于从模块到子网络再到完整网络的概念设计出来的。一阶沙漏网络是最简单的旁路相加架构，上半路维持原有尺度，下半路经历下采样再进行上采样。而二阶沙漏网络则在一阶架构上嵌入一阶网络。至于之后的高阶沙漏网络，则是一层一层嵌套，从本质上说，子模块都是一样的。\n",
    "相对于DeepPose直接将人体姿态估计定位成回归问题并将AlexNet改造成对应结构，堆叠沙漏网络显然在姿态估计这个任务上走得更深。堆叠沙漏网络通过多次自下而上过程和自上而下过程来对全局信息和局部信息进行融合，且通过中继监督提升了整个网络的预测精度，同时拥有了能够纠正初始预测错误的能力。如果最初的估计与实际情况相差甚远，那么后续的迭代无法纠正初始的错误。\n",
    "相较于回归坐标，回归热图的优势有：\n",
    "* 直接坐标回归，网络需要学习的函数高度非线性，不适合网络学习；\n",
    "* 如果直接坐标回归，人体特征点坐标的数值范围较大，不适合网络学习；\n",
    "* 比起坐标的只有两个数，热图能够提供空间上更丰富的监督信息。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.4 小结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在本章中，我们学习了一些经典的人体姿态估计方法，动手学习了DeepPose以及堆叠沙漏网络，并在LSP数据集上进行了训练与测试。人体姿态估计是计算机视觉中一个基础问题，只有准确地估计了图像中人体的姿态，才能赋能后续的操作如电影捕捉、动画制作、轨迹跟踪、人机交互以及运动分析等。接下来我们将继续学习如何从视频中识别运动。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5 参考文献\n",
    "\n",
    "[1] JOHNSON S, EVERINGHAM M. Clustered pose and nonlinear appearance models for human pose estimation[C]//BMVC. 2010, 2(4): 5.\n",
    "\n",
    "[2] SAPP B, TASKAR B. MODEC: Multimodal decomposable models for human pose estimation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013: 3674-3681.\n",
    "\n",
    "[3] ANDRILOUKA M, PISCHCHULIN L, GEHLER P, et al. 2D human pose estimation: New benchmark and state of the art analysis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014: 3686-3693.\n",
    "\n",
    "[4] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: Common Objects in Context[C]//Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer International Publishing, 2014: 740-755.\n",
    "\n",
    "[5] TOSHEV A, SZEGEDY C. Deeppose: Human pose estimation via deep neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014: 1653-1660.\n",
    "\n",
    "[6] BULAT A, TZIMIROPOULOS G. Human pose estimation via convolutional part heatmap regression[C]//Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14. Springer International Publishing, 2016: 717-732.\n",
    "\n",
    "[7] WEI S E, RAMAKRISHNA V, KANADE T, et al. Convolutional pose machines[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 4724-4732.\n",
    "\n",
    "[8] CARREIRA J, AGRAWAL P, FRAGKIADAKI K, et al. Human pose estimation with iterative error feedback[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 4733-4742.\n",
    "\n",
    "[9] NEWELL A, YANG K, DENG J. Stacked hourglass networks for human pose estimation[C]//Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14. Springer International Publishing, 2016: 483-499."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- 目前的困难：\n",
    "1. 姿态估计的精确度问题：由于人体姿态估计系统使用的是多种视觉信息，如果它们不能够准确地追踪到人体的每个部位，那么估计出来的结果也会不准确。\n",
    "1. 尺度不变性问题：尽管现有的人体姿态估计算法可以处理不同尺度的图像，但它们仍然存在尺度不变性问题，即当姿态发生变化时，估计的结果可能会受到影响。\n",
    "1. 光照变化问题：由于光照变化会对视觉信息产生影响，因此人体姿态估计系统可能无法准确地识别不同光照环境下的人体部位。\n",
    "1. 复杂背景问题：复杂的背景可能会干扰人体姿态估计系统的性能，使得它无法准确地识别人体部位。 -->"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
