{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "59f0b8d2",
   "metadata": {
    "papermill": {
     "duration": 0.005681,
     "end_time": "2023-08-15T09:00:42.167660",
     "exception": false,
     "start_time": "2023-08-15T09:00:42.161979",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 代码实现"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37573ecf",
   "metadata": {
    "papermill": {
     "duration": 0.00558,
     "end_time": "2023-08-15T09:00:42.179143",
     "exception": false,
     "start_time": "2023-08-15T09:00:42.173563",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "在上面的章节中，我们介绍了双目照相机的构造以及其计算深度的原理，接下来我们进入动手学环节，学习如何从一组双目图像中获得视差图。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "62cf8777",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-15T09:00:42.194589Z",
     "iopub.status.busy": "2023-08-15T09:00:42.194082Z",
     "iopub.status.idle": "2023-08-15T09:00:44.888956Z",
     "shell.execute_reply": "2023-08-15T09:00:44.887420Z"
    },
    "papermill": {
     "duration": 2.70709,
     "end_time": "2023-08-15T09:00:44.892116",
     "exception": false,
     "start_time": "2023-08-15T09:00:42.185026",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'learncv_img'...\n",
      "remote: Enumerating objects: 89, done.\u001b[K\n",
      "remote: Counting objects: 100% (8/8), done.\u001b[K\n",
      "remote: Compressing objects: 100% (8/8), done.\u001b[K\n",
      "remote: Total 89 (delta 0), reused 0 (delta 0), pack-reused 81\u001b[K\n",
      "Receiving objects: 100% (89/89), 14.23 MiB | 1023.00 KiB/s, done.\n",
      "Resolving deltas: 100% (10/10), done.\n"
     ]
    }
   ],
   "source": [
    "# 首先clone对应仓库\n",
    "! git clone https://github.com/boyu-ai/Hands-on-CV.git"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fab2a0a4",
   "metadata": {},
   "source": [
    "首先我们读取左右两张图像，并做一个简单的可视化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6813f9b2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-15T09:00:44.909845Z",
     "iopub.status.busy": "2023-08-15T09:00:44.909028Z",
     "iopub.status.idle": "2023-08-15T09:00:45.593286Z",
     "shell.execute_reply": "2023-08-15T09:00:45.592411Z"
    },
    "papermill": {
     "duration": 0.698061,
     "end_time": "2023-08-15T09:00:45.597860",
     "exception": false,
     "start_time": "2023-08-15T09:00:44.899799",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 216532 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os\n",
    "import time\n",
    "import cv2 as cv\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# 获取左照相机图像\n",
    "limg_raw = np.asanyarray(Image.open(\n",
    "    r\"./learncv_img/stereo/scene1.row3.col1.ppm\"))   \n",
    " \n",
    "# 获取右照相机图像\n",
    "rimg_raw = np.asanyarray(Image.open(\n",
    "    r\"./learncv_img/stereo/scene1.row3.col3.ppm\"))  \n",
    "  \n",
    "# 将图像转化为灰度图\n",
    "limg = cv.cvtColor(limg_raw, cv.COLOR_BGR2GRAY)     \n",
    "rimg = cv.cvtColor(rimg_raw, cv.COLOR_BGR2GRAY)\n",
    "\n",
    "# 将图像转化为double类型\n",
    "limg = np.asanyarray(limg, dtype=np.double)     \n",
    "rimg = np.asanyarray(rimg, dtype=np.double)\n",
    "\n",
    "# 定义一个和输入图像大小相同的数组\n",
    "img_size = np.shape(limg)[0:2]   \n",
    "\n",
    "# 简单可视化左右图像\n",
    "plt.figure(dpi=100)                             \n",
    "plt.title(\"binocular images\")\n",
    "plt.imshow(np.hstack([limg_raw, rimg_raw]))\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fb1cbe6",
   "metadata": {},
   "source": [
    "接下来，我们实现生成视差图的函数。此函数首先计算左右图像的整体差距，然后对每个像素进行视差搜索，并选择最小匹配代价所对应的视差作为最终视差值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a4b74df",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-15T09:00:45.621841Z",
     "iopub.status.busy": "2023-08-15T09:00:45.620858Z",
     "iopub.status.idle": "2023-08-15T09:01:12.803328Z",
     "shell.execute_reply": "2023-08-15T09:01:12.802133Z"
    },
    "papermill": {
     "duration": 27.198257,
     "end_time": "2023-08-15T09:01:12.806815",
     "exception": false,
     "start_time": "2023-08-15T09:00:45.608558",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "所用时间: 11.269958972930908\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 144388 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def get_disp_map(max_D, win_size, img_size, match_mode=\"SAD\"):\n",
    "    \"\"\"\n",
    "    根据左右图像，获取视差图。\n",
    "    \n",
    "    参数:\n",
    "    - max_D: 左右图像之间最大可能的视差值。\n",
    "    - win_size: 块尺寸。\n",
    "    - img_size: 双目图像尺寸。\n",
    "    - match_mode: 匹配模式，可选\"SAD\"或\"SSD\"。\n",
    "    \"\"\"\n",
    "    # 记录初始时间\n",
    "    t1 = time.time()  \n",
    "\n",
    "    # 初始化视差图                     \n",
    "    imgDiff = np.zeros((img_size[0], img_size[1], max_D))   \n",
    "\n",
    "    def SAD(x, y, error_map, win_size):\n",
    "        return np.sum(error_map[(x - win_size):(x + win_size), \n",
    "                                (y - win_size):(y + win_size)])\n",
    "\n",
    "    def SSD(x, y, error_map, win_size):\n",
    "        return np.sum(np.square(error_map[\n",
    "        (x - win_size):(x + win_size), (y - win_size):(y + win_size)]))\n",
    "\n",
    "    # 遍历所有可能的视差值\n",
    "    for i in range(0, max_D):\n",
    "        # 记录视差为i时，左右图像的整体差距\n",
    "        e = np.abs(rimg[:, 0:(img_size[1]-i)] - limg[:, i:img_size[1]]) \n",
    "        e2 = np.zeros(img_size)\n",
    "\n",
    "        # 遍历所有像素    \n",
    "        for x in range(0, img_size[0]):\n",
    "            for y in range(0, img_size[1]):\n",
    "                # 根据匹配模式计算匹配代价\n",
    "                if match_mode == \"SAD\":\n",
    "                    # 计算SAD值\n",
    "                    e2[x, y] = SAD(x, y, e, win_size) \n",
    "                elif match_mode == \"SSD\":\n",
    "                    # 计算SSD值\n",
    "                    e2[x, y] = SSD(x, y, e, win_size) \n",
    "                else:\n",
    "                    raise NotImplemented\n",
    "        \n",
    "        # 将匹配代价存储到imgDiff中  \n",
    "        imgDiff[:, :, i] = e2\n",
    "\n",
    "    # 初始化视差图\n",
    "    dispMap = np.zeros(img_size)    \n",
    "    for x in range(0, img_size[0]):\n",
    "        for y in range(0, img_size[1]):\n",
    "            # 对匹配代价进行排序\n",
    "            val = np.sort(imgDiff[x, y, :])  \n",
    "            \n",
    "            # 简单判断是否出了边界\n",
    "            if np.abs(val[0]-val[1]) > 10:     \n",
    "                # 找到最小匹配代价的视差索引\n",
    "                val_id = np.argsort(imgDiff[x, y, :])      \n",
    "                # 将视差值映射到 0-255 之间\n",
    "                dispMap[x, y] = val_id[0] / max_D * 255        \n",
    "\n",
    "    # 打印运行时间\n",
    "    print(\"所用时间:\", time.time() - t1)\n",
    "\n",
    "    # 显示视差图\n",
    "    plt.figure(\"视差图\")\n",
    "    plt.imshow(dispMap)\n",
    "    plt.show()\n",
    "\n",
    "    # 返回视差图\n",
    "    return dispMap\n",
    "    \n",
    "# 最大视差\n",
    "max_D = 25   \n",
    "# 滑动窗口大小\n",
    "win_size = 5         \n",
    "# 可视化SAD匹配效果，块尺寸为5像素\n",
    "_ = get_disp_map(max_D, win_size, img_size) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e12919e9",
   "metadata": {
    "papermill": {
     "duration": 0.012862,
     "end_time": "2023-08-15T09:01:12.832870",
     "exception": false,
     "start_time": "2023-08-15T09:01:12.820008",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "通过生成双目图像的视差图，我们可以观察到一些直观的现象，比如台灯在视差图中的亮度较高，这表明其视差值相对较大；而背景部分的亮度较低，表示视差值较小。视差图中的亮度变化直接反映了物体与相机之间的相对深度差异：亮度较高的区域（视差大）意味着物体相对较近，亮度较低的区域（视差小）则表明物体相对较远。因此，利用视差图，我们可以有效地估计场景中各个物体的深度信息。\n",
    "\n",
    "接下来我们看看调大块匹配的尺寸的效果:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "de901c9a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-15T09:01:12.861527Z",
     "iopub.status.busy": "2023-08-15T09:01:12.860716Z",
     "iopub.status.idle": "2023-08-15T09:01:47.395885Z",
     "shell.execute_reply": "2023-08-15T09:01:47.394536Z"
    },
    "papermill": {
     "duration": 34.552547,
     "end_time": "2023-08-15T09:01:47.398581",
     "exception": false,
     "start_time": "2023-08-15T09:01:12.846034",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "所用时间: 14.947252035140991\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 36632 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 测试更大的块尺寸\n",
    "win_size = 30 \n",
    "# SAD匹配效果，块尺寸为30像素\n",
    "_ = get_disp_map(max_D, win_size, img_size) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6983cdf9",
   "metadata": {
    "papermill": {
     "duration": 0.013375,
     "end_time": "2023-08-15T09:01:47.425643",
     "exception": false,
     "start_time": "2023-08-15T09:01:47.412268",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "不难发现，块尺寸越大，预测出的视差图更加平滑，细节丢失更加严重，同时计算的时间也增加了。\n",
    "\n",
    "接下来我们来看看SSD的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "43f37d7b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-15T09:01:47.454620Z",
     "iopub.status.busy": "2023-08-15T09:01:47.453943Z",
     "iopub.status.idle": "2023-08-15T09:02:21.159150Z",
     "shell.execute_reply": "2023-08-15T09:02:21.157996Z"
    },
    "papermill": {
     "duration": 33.72327,
     "end_time": "2023-08-15T09:02:21.162282",
     "exception": false,
     "start_time": "2023-08-15T09:01:47.439012",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "所用时间: 13.194314956665039\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 119600 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "win_size = 5\n",
    "# SSD匹配效果，块尺寸为5像素\n",
    "_ = get_disp_map(max_D, win_size, img_size, match_mode=\"SSD\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "86f72cf9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "所用时间: 18.293694019317627\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 37036 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "win_size = 30\n",
    "# SSD匹配效果，块尺寸为30像素\n",
    "_ = get_disp_map(max_D, win_size, img_size, match_mode=\"SSD\") "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ee51e83",
   "metadata": {
    "papermill": {
     "duration": 0.015168,
     "end_time": "2023-08-15T09:02:21.192996",
     "exception": false,
     "start_time": "2023-08-15T09:02:21.177828",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "对比SAD和SSD的效果，我们可以发现，SSD保留了更多细节，但是同时它也花费了更多的时间。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 112.65169,
   "end_time": "2023-08-15T09:02:22.194127",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-08-15T09:00:29.542437",
   "version": "2.4.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
