# inspect_image

> Send a local image file to a vision-capable model and return text analysis.

## Source
- Entry: `packages/coding-agent/src/tools/inspect-image.ts`
- Model-facing prompt: `packages/coding-agent/src/prompts/tools/inspect-image.md`
- Key collaborators:
  - `packages/coding-agent/src/tools/inspect-image-renderer.ts` — TUI call/result rendering.
  - `packages/coding-agent/src/utils/image-loading.ts` — path resolution, type detection, size gate, optional resize.
  - `packages/coding-agent/src/utils/image-resize.ts` — downscale and recompress oversized images.
  - `packages/coding-agent/src/tools/path-utils.ts` — resolve input path relative to session cwd.
  - `packages/utils/src/mime.ts` — detect supported image formats from file bytes.

## Inputs

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `path` | `string` | Yes | Image path passed to `loadImageInput`; resolved relative to `session.cwd` by `resolveReadPath(...)`. |
| `question` | `string` | Yes | User prompt sent as a text content block alongside the image. |

## Outputs
The tool returns a single `AgentToolResult`:

- `content`: one text block, `[{ type: "text", text }]`, where `text` is the concatenated assistant text content from the model response.
- `details`:
  - `model`: `<provider>/<id>` of the selected model.
  - `imagePath`: resolved filesystem path returned by `loadImageInput(...)`.
  - `mimeType`: MIME type actually sent to the model after optional resize/re-encode.

Model-visible output is single-shot, not streamed by this tool.

TUI rendering adds presentation-only truncation from `packages/coding-agent/src/tools/inspect-image-renderer.ts`:

- call preview truncates `question` to 100 columns,
- result view shows 4 lines collapsed or 16 lines expanded,
- each rendered output line is truncated to 120 columns,
- footer metadata shows `model · mimeType` when present.

## Flow
1. `InspectImageTool.execute(...)` rejects immediately if `images.blockImages` is enabled in session settings.
2. It reads `session.modelRegistry`; missing registry, empty registry, missing API key, or unresolved model each raise `ToolError` from `packages/coding-agent/src/tools/inspect-image.ts`.
3. Model selection tries, in order, `@vision`, `@default`, the active model string from the session, then `availableModels[0]`. `expandRoleAlias(...)` and `resolveModelFromString(...)` handle each lookup.
4. The chosen model must advertise `input.includes("image")`; otherwise execution fails before reading the file.
5. `loadImageInput(...)` in `packages/coding-agent/src/utils/image-loading.ts` resolves the path with `resolveReadPath(...)`, detects MIME type with `readImageMetadata(...)`, and rejects files larger than `MAX_IMAGE_INPUT_BYTES` (`20 * 1024 * 1024`, 20 MiB) using `ImageInputTooLargeError`.
6. `readImageMetadata(...)` in `packages/utils/src/mime.ts` inspects file headers only. Supported detected MIME types are `image/png`, `image/jpeg`, `image/gif`, and `image/webp`.
7. `loadImageInput(...)` is called with `excludeWebP: webpExclusionForModel(model)` (`true` only for models that cannot decode WebP, e.g. the Ollama family). It calls `resizeImage(...)` when `images.autoResize` is true, or when `excludeWebP` is set and the detected type is `image/webp` — re-encoding away from WebP even with auto-resize off. The `excludeWebP` flag is forwarded into `resizeImage(...)`. Resize failures are swallowed there and the original bytes are kept.
8. If MIME detection returned no supported image type, `execute(...)` throws `ToolError("inspect_image only supports PNG, JPEG, GIF, and WEBP files detected by file content.")`.
9. The tool calls `instrumentedCompleteSimple(...)` with one user message containing two content parts in order:
   - `{ type: "image", data: imageInput.data, mimeType: imageInput.mimeType }`
   - `{ type: "text", text: params.question }`
10. `systemPrompt` is a one-element array rendered from `packages/coding-agent/src/prompts/tools/inspect-image-system.md`; telemetry is tagged with oneshot kind `inspect_image`.
11. If the model response stop reason is `error` or `aborted`, the tool maps that to `ToolError`.
12. `extractTextContent(...)` from `packages/coding-agent/src/commit/utils.ts` concatenates only `text` content blocks from the assistant message, trims the result, and the tool fails if nothing remains.
13. Success returns the text plus `details`; `inspectImageToolRenderer` formats the result for the TUI.

## Modes / Variants
- **Original image path**: `images.autoResize` disabled. The original file bytes are base64-encoded and sent with the detected MIME type.
- **Auto-resized path**: `images.autoResize` enabled. `resizeImage(...)` may downscale and re-encode the image before upload.
- **Unsupported image path**: file exists but header sniffing does not identify PNG/JPEG/GIF/WEBP. The tool returns a `ToolError` before any model call.
- **Oversize image path**: file size exceeds 20 MiB before upload. The tool returns a `ToolError` before any model call.

## Side Effects
- Filesystem
  - Resolves and reads the target image from disk.
  - Stats the file once with `Bun.file(...).stat()` and reads it fully with `fs.readFile(...)`.
- Network
  - Sends the final base64 image payload plus question text to the selected model through `instrumentedCompleteSimple(...)` / the configured simple completion implementation.
- Session state
  - Reads session settings, active model preferences, cwd, and model registry.
- Background work / cancellation
  - Passes the caller `AbortSignal` into `instrumentedCompleteSimple(...)` and the configured simple completion implementation.
  - Image preprocessing is local and not cancellation-aware in these helpers.

## Limits & Caps
- Supported detected input formats: `image/png`, `image/jpeg`, `image/gif`, `image/webp` (`SUPPORTED_IMAGE_MIME_TYPES` in `packages/utils/src/mime.ts`).
- Metadata sniff cap: `DEFAULT_IMAGE_METADATA_HEADER_BYTES = 256 * 1024` bytes. Format detection only reads up to 256 KiB from the file header.
- Availability is gated by `inspect_image.enabled`, default `false`, in `packages/coding-agent/src/config/settings-schema.ts` / `packages/coding-agent/src/tools/index.ts`.
- Upload input cap: `MAX_IMAGE_INPUT_BYTES = 20 * 1024 * 1024` bytes (20 MiB) in `packages/coding-agent/src/utils/image-loading.ts`.
- Auto-resize defaults in `packages/coding-agent/src/utils/image-resize.ts`:
  - `maxWidth: 1568`
  - `maxHeight: 1568`
  - `maxBytes: 500 * 1024` bytes (500 KiB target)
  - `jpegQuality: 80`
- Resize fast path: if the original image is already within `1568x1568` and within `maxBytes / 4` (125 KiB by default), `resizeImage(...)` returns the original bytes unchanged.
- Resize quality ladder: after the first encode pass, lossy retries use qualities `[70, 60, 50, 40]`.
- Resize dimension ladder: if quality reduction still misses the byte target, retries scale dimensions by `[1.0, 0.75, 0.5, 0.35, 0.25]` and stop if either dimension would fall below `100` pixels.
- First resize pass encodes PNG, JPEG, and WebP, then keeps the smallest encoded buffer. Fallback passes encode JPEG and WebP only, again keeping the smaller output. WebP is excluded from both ladders when `OMP_NO_WEBP=1`/`true` (or `excludeWebP` is passed).
- Renderer caps:
  - `INSPECT_QUESTION_PREVIEW_WIDTH = 100`
  - `INSPECT_OUTPUT_COLLAPSED_LINES = 4`
  - `INSPECT_OUTPUT_EXPANDED_LINES = 16`
  - `INSPECT_OUTPUT_LINE_WIDTH = 120`

## Errors
- Settings gate:
  - `Image submission is disabled by settings (images.blockImages=true). Disable it to use inspect_image.`
- Model resolution / capability:
  - `Model registry is unavailable for inspect_image.`
  - `No models available for inspect_image.`
  - `Unable to resolve a model for inspect_image.`
  - `Resolved model <provider>/<id> does not support image input. Configure a vision-capable model for modelRoles.vision.`
  - `No API key available for <provider>/<id>. Configure credentials for this provider or choose another vision-capable model.`
- Input file:
  - `Image file too large: <size> exceeds <limit> limit.` from `ImageInputTooLargeError`, remapped to `ToolError`.
  - `inspect_image only supports PNG, JPEG, GIF, and WEBP files detected by file content.` when header sniffing fails.
- Model call:
  - `inspect_image request failed.` if the response stop reason is `error` without a provider message.
  - Provider `errorMessage` is passed through when present.
  - `inspect_image request aborted.` on aborted responses.
  - `inspect_image model returned no text output.` when the assistant message contains no text blocks after filtering.

Failures surface as thrown `ToolError`s from `execute(...)`; the normal success return shape is not used for error reporting.

## Notes
- The tool schema is not marked strict in `InspectImageTool`; callers should still treat only `path` and `question` as supported inputs because the implementation reads no other fields.
- The model-facing prompt path on disk is `packages/coding-agent/src/prompts/tools/inspect-image.md`; the assignment's underscore form does not exist.
- Format support is based on file content, not filename extension. Renaming a non-image file to `.png` does not make it valid.
- `resolveReadPath(...)` tries macOS-specific path variants: shell-unescaped spaces, AM/PM narrow no-break-space filenames, NFD normalization, and curly-quote variants.
- `loadImageInput(...)` also computes `textNote`, `dimensionNote`, and final `bytes`, but `inspect_image` does not include those in tool output.
- Auto-resize can change the MIME type sent to the model. A JPEG or GIF input may be uploaded as PNG, JPEG, or WebP depending on which encoder output is smallest.
- If `resizeImage(...)` throws or cannot decode the image, `loadImageInput(...)` silently keeps the original base64 payload instead of failing.
