From a036d985b769b071a09b8f60f160b3bd03ca7fe6 Mon Sep 17 00:00:00 2001 From: M5Max128 Date: Wed, 27 May 2026 14:35:53 +0800 Subject: [PATCH] docs: add Thumbnail QA Analysis for M5Max48 implementation --- docs_v1.0/DESIGN/Thumbnail_QA_Analysis.md | 340 ++++++++++++++++++++++ 1 file changed, 340 insertions(+) create mode 100644 docs_v1.0/DESIGN/Thumbnail_QA_Analysis.md diff --git a/docs_v1.0/DESIGN/Thumbnail_QA_Analysis.md b/docs_v1.0/DESIGN/Thumbnail_QA_Analysis.md new file mode 100644 index 0000000..015bb75 --- /dev/null +++ b/docs_v1.0/DESIGN/Thumbnail_QA_Analysis.md @@ -0,0 +1,340 @@ +--- +title: Thumbnail Endpoint Quality Assurance Analysis +version: 1.0.0 +date: 2026-05-27 +author: M5Max128 +status: research_complete +--- + +# Thumbnail Endpoint Quality Assurance Analysis + +## Scope + +| Item | Status | +|------|--------| +| Research | Complete | +| Implementation | Pending (M5Max48) | +| Affected Endpoints | 2 | + +## Overview + +Thumbnail endpoints currently lack quality validation, resulting in potential anomalies: +- **Empty images** - ffmpeg produces 0 bytes output +- **Black frames** - extracted frame is all black +- **Corrupted JPEG** - incomplete ffmpeg output + +## Affected Endpoints + +| Endpoint | File | Line | +|----------|------|------| +| `/api/v1/file/:file_uuid/thumbnail` | `src/api/media_api.rs` | 700-764 | +| `/api/v1/file/:file_uuid/trace/:trace_id/thumbnail` | `src/api/trace_agent_api.rs` | 514-556 | + +--- + +## Anomaly Classification + +### Type 1: Empty Image (No Frame) + +**Symptom**: Returns 0 bytes or very small JPEG + +**Root Causes**: +1. `frame_number > total_frames` - requested frame exceeds video length +2. Video file missing or corrupted +3. Codec does not support frame-level seek +4. ffmpeg `-vf select` filter finds no matching frame + +**Code Locations**: +- `media_api.rs:710-716` - `query_auto_representative_frame()` may return invalid frame +- `media_api.rs:720-728` - `file_path` query may return non-existent file +- `media_api.rs:754-756` - only checks `output.status.success()`, not output content + +### Type 2: Black Frame + +**Symptom**: Returns valid JPEG but all black or very dark + +**Root Causes**: +1. `crop` parameters exceed video dimensions (`x+w > width` or `y+h > height`) +2. Extracted frame is from fade-in/fade-out transition +3. Video has black opening/closing credits +4. Low-light scene + +**Code Locations**: +- `media_api.rs:731-735` - crop validation missing +- `trace_agent_api.rs:530` - crop may exceed dimensions + +### Type 3: Corrupted JPEG + +**Symptom**: Returns incomplete JPEG (browser shows broken image) + +**Root Causes**: +1. ffmpeg stdout pipe interrupted before completion +2. ffmpeg process killed mid-output +3. JPEG encoder failure +4. Incomplete write to stdout buffer + +**Code Locations**: +- `media_api.rs:751` - pipe output may be truncated +- `media_api.rs:758-763` - no JPEG validation before serving + +--- + +## Current Quality Mechanisms + +### Endpoint 1: `face_thumbnail` + +| Mechanism | Status | Location | +|-----------|--------|----------| +| Representative frame selection | Present | `tkg::query_auto_representative_frame()` | +| ffmpeg success check | Present | `output.status.success()` | +| JPEG validation | Missing | - | +| Size validation | Missing | - | +| Black frame detection | Missing | - | +| Retry mechanism | Missing | - | + +### Endpoint 2: `get_trace_thumbnail` + +| Mechanism | Status | Location | +|-----------|--------|----------| +| Blur detection (candidate selection) | Present | `select_rep_face()` lines 463-480 | +| Confidence filter (>0.7) | Present | `select_rep_face()` line 429 | +| QC metadata filter | Present | `select_rep_face()` line 430 | +| ffmpeg success check | Present | `status.status.success()` | +| JPEG validation | Missing | - | +| Black frame detection (extraction) | Missing | - | +| Retry mechanism | Missing | - | + +**Note**: `select_rep_face()` has sophisticated quality control for SELECTING the representative face, but the actual EXTRACTION step lacks validation. + +--- + +## Root Cause Analysis + +### A. Input Data Problems + +| Problem | Impact | Condition | +|---------|--------|-----------| +| `frame_number > total_frames` | Empty image | TKG returns wrong frame, user passes invalid value | +| `crop exceeds dimensions` | Black frame / error | face bbox incorrect, video resolution changed | +| Video file missing | 500 error | File deleted/moved | +| Codec不支持seek | Empty/corrupted | Some codecs only support sequential read | + +### B. ffmpeg Execution Problems + +| Problem | Impact | Cause | +|---------|--------|-------| +| `select` no output | Empty JPEG | frame超出範圍 → ffmpeg skips all frames | +| Pipe interrupted | Corrupted JPEG | stdout buffer full, ffmpeg terminated early | +| `-ss` imprecise | Wrong frame | input seeking approximate, error ±5 frames | +| crop failure | Black frame / 500 | `x+w > width` or `y+h > height` | + +### C. Quality Control Gaps + +| Gap | Impact | Current | +|-----|--------|---------| +| No JPEG validation | Corrupted image served | Only checks exit code | +| No size check | 0 bytes returned | No output length check | +| No black detection | Black frame served | blurdetect only in candidate selection | +| No retry | Single failure = error | No retry mechanism | + +--- + +## Concrete Failure Cases + +### Case 1: Frame Exceeds Range + +``` +Video: total_frames=1000 (DB record) +Actual: video has only 950 frames (file truncated) +Request: frame=980 +ffmpeg: select=eq(n\,980) → no match +Output: 0 bytes JPEG +Frontend: blank image +``` + +### Case 2: Crop Exceeds Dimensions + +``` +Video: 1920x1080 +face_bbox: x=1850, y=1050, w=100, h=100 +ffmpeg: crop=100:100:1850:1050 +Result: x+100=1950 > 1920 → ffmpeg error or black border +``` + +### Case 3: Seek Imprecise + +``` +Video: 25fps +Request: frame=1000 (40 seconds) +ffmpeg -ss 40.0 -i video +Actual: seeks to frame 995~1005 range +Result: extracts different face than select_rep_face chose +``` + +### Case 4: Pipe Interrupted + +``` +ffmpeg -i large_video -vf select=eq(n\,50000) -f image2pipe - +Video large, select needs scan to frame 50000 +Pipe buffer full → ffmpeg may be killed or terminate early +Output: incomplete JPEG (missing FFD9 footer) +``` + +--- + +## Recommended Fixes + +### Phase P0: Critical (Must Implement) + +| Fix | Description | LOC | Location | +|-----|-------------|-----|----------| +| **Frame validation** | `frame <= total_frames` | ~20 | `media_api.rs:707-718` | +| **Crop validation** | `x+w <= width, y+h <= height` | ~15 | `media_api.rs:731-735` | +| **JPEG header check** | `data[0..3] == [0xFF,0xD8,0xFF]` | ~10 | Helper function | +| **JPEG footer check** | `data[-2..] == [0xFF,0xD9]` | ~10 | Helper function | +| **Minimum size check** | `data.len() > 100` | ~5 | Helper function | + +### Phase P1: Important (Should Implement) + +| Fix | Description | LOC | Location | +|-----|-------------|-----|----------| +| **Black frame detection** | ffmpeg `-vf blackdetect` filter | ~30 | After extraction | +| **Output seeking** | Move `-ss` after `-i` for precision | ~5 | `trace_agent_api.rs:527` | + +### Phase P2: Enhancement (Nice to Have) + +| Fix | Description | LOC | Location | +|-----|-------------|-----|----------| +| **Retry mechanism** | Max 3 attempts, offset +30 frames each | ~50 | Both endpoints | +| **Fallback frame** | Extract middle frame if all fail | ~30 | Both endpoints | + +--- + +## Implementation Plan + +### Step 1: Create Validation Module + +Create `src/core/thumbnail/validator.rs`: + +```rust +pub fn validate_jpeg(data: &[u8]) -> Result<()> { + // P0-1: Minimum size + if data.len() < 100 { + bail!("JPEG too small: {} bytes", data.len()); + } + + // P0-2: JPEG header (SOI marker) + if data[0..3] != [0xFF, 0xD8, 0xFF] { + bail!("Invalid JPEG header"); + } + + // P0-3: JPEG footer (EOI marker) + if data[data.len()-2..] != [0xFF, 0xD9] { + bail!("Incomplete JPEG"); + } + + Ok(()) +} +``` + +### Step 2: Add Frame/Crop Validation + +In `media_api.rs`: + +```rust +// P0-4: Validate frame number +let total_frames: i64 = sqlx::query_scalar(...) + .bind(&file_uuid) + .fetch_one(pool) + .await?; + +if frame > total_frames { + return Err(StatusCode::BAD_REQUEST); +} + +// P0-5: Validate crop dimensions +if let (Some(x), Some(y), Some(w), Some(h)) = (q.x, q.y, q.w, q.h) { + let (width, height): (i32, i32) = sqlx::query_as(...) + .bind(&file_uuid) + .fetch_one(pool) + .await?; + + if x + w > width || y + h > height { + return Err(StatusCode::BAD_REQUEST); + } +} +``` + +### Step 3: Integrate Validation + +In both endpoints, after ffmpeg extraction: + +```rust +// Apply validation +validate_jpeg(&output.stdout) + .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?; +``` + +--- + +## Testing Strategy + +### Test Cases + +| Test | Input | Expected | +|------|-------|----------| +| Valid frame | `frame=500` (valid) | JPEG returned | +| Frame exceeds | `frame=999999` | 400 BAD_REQUEST | +| Valid crop | `x=100,y=100,w=200,h=200` | JPEG returned | +| Crop exceeds | `x=1800,y=1000,w=200,h=200` | 400 BAD_REQUEST | +| Empty video | corrupted video file | 500 INTERNAL_ERROR | +| Black frame | fade-out frame | Retry or fallback | + +--- + +## Files to Modify + +| File | Changes | +|------|---------| +| `src/core/thumbnail/mod.rs` | Add validator module | +| `src/core/thumbnail/validator.rs` | New file (validation helpers) | +| `src/api/media_api.rs` | Add validation in `face_thumbnail()` | +| `src/api/trace_agent_api.rs` | Add validation in `get_trace_thumbnail()` | + +--- + +## Estimated Effort + +| Phase | LOC | Time | +|-------|-----|------| +| P0 (Critical) | ~60 | 1-2 days | +| P1 (Important) | ~35 | 1 day | +| P2 (Enhancement) | ~80 | 2-3 days | +| **Total** | ~175 | 4-6 days | + +--- + +## Version History + +| Version | Date | Author | Changes | +|---------|------|--------|---------| +| 1.0.0 | 2026-05-27 | M5Max128 | Initial analysis complete | + +--- + +## Next Steps for M5Max48 + +1. Read this document +2. Implement P0 fixes first +3. Test with edge cases +4. Add P1/P2 as needed +5. Update `AGENTS.md` if adding new validation commands + +--- + +## References + +- `docs_v1.0/DESIGN/Processor_Refactoring_Assessment.md` - Processor refactoring priorities +- `src/api/media_api.rs:700-764` - face_thumbnail implementation +- `src/api/trace_agent_api.rs:394-556` - select_rep_face and get_trace_thumbnail +- `ffmpeg -vf blackdetect` documentation \ No newline at end of file