docs: add Thumbnail QA Analysis for M5Max48 implementation
This commit is contained in:
M5Max128
340
docs_v1.0/DESIGN/Thumbnail_QA_Analysis.md
Normal file
340
docs_v1.0/DESIGN/Thumbnail_QA_Analysis.md
Normal file
@@ -0,0 +1,340 @@
|
||||
---
|
||||
title: Thumbnail Endpoint Quality Assurance Analysis
|
||||
version: 1.0.0
|
||||
date: 2026-05-27
|
||||
author: M5Max128
|
||||
status: research_complete
|
||||
---
|
||||
|
||||
# Thumbnail Endpoint Quality Assurance Analysis
|
||||
|
||||
## Scope
|
||||
|
||||
| Item | Status |
|
||||
|------|--------|
|
||||
| Research | Complete |
|
||||
| Implementation | Pending (M5Max48) |
|
||||
| Affected Endpoints | 2 |
|
||||
|
||||
## Overview
|
||||
|
||||
Thumbnail endpoints currently lack quality validation, resulting in potential anomalies:
|
||||
- **Empty images** - ffmpeg produces 0 bytes output
|
||||
- **Black frames** - extracted frame is all black
|
||||
- **Corrupted JPEG** - incomplete ffmpeg output
|
||||
|
||||
## Affected Endpoints
|
||||
|
||||
| Endpoint | File | Line |
|
||||
|----------|------|------|
|
||||
| `/api/v1/file/:file_uuid/thumbnail` | `src/api/media_api.rs` | 700-764 |
|
||||
| `/api/v1/file/:file_uuid/trace/:trace_id/thumbnail` | `src/api/trace_agent_api.rs` | 514-556 |
|
||||
|
||||
---
|
||||
|
||||
## Anomaly Classification
|
||||
|
||||
### Type 1: Empty Image (No Frame)
|
||||
|
||||
**Symptom**: Returns 0 bytes or very small JPEG
|
||||
|
||||
**Root Causes**:
|
||||
1. `frame_number > total_frames` - requested frame exceeds video length
|
||||
2. Video file missing or corrupted
|
||||
3. Codec does not support frame-level seek
|
||||
4. ffmpeg `-vf select` filter finds no matching frame
|
||||
|
||||
**Code Locations**:
|
||||
- `media_api.rs:710-716` - `query_auto_representative_frame()` may return invalid frame
|
||||
- `media_api.rs:720-728` - `file_path` query may return non-existent file
|
||||
- `media_api.rs:754-756` - only checks `output.status.success()`, not output content
|
||||
|
||||
### Type 2: Black Frame
|
||||
|
||||
**Symptom**: Returns valid JPEG but all black or very dark
|
||||
|
||||
**Root Causes**:
|
||||
1. `crop` parameters exceed video dimensions (`x+w > width` or `y+h > height`)
|
||||
2. Extracted frame is from fade-in/fade-out transition
|
||||
3. Video has black opening/closing credits
|
||||
4. Low-light scene
|
||||
|
||||
**Code Locations**:
|
||||
- `media_api.rs:731-735` - crop validation missing
|
||||
- `trace_agent_api.rs:530` - crop may exceed dimensions
|
||||
|
||||
### Type 3: Corrupted JPEG
|
||||
|
||||
**Symptom**: Returns incomplete JPEG (browser shows broken image)
|
||||
|
||||
**Root Causes**:
|
||||
1. ffmpeg stdout pipe interrupted before completion
|
||||
2. ffmpeg process killed mid-output
|
||||
3. JPEG encoder failure
|
||||
4. Incomplete write to stdout buffer
|
||||
|
||||
**Code Locations**:
|
||||
- `media_api.rs:751` - pipe output may be truncated
|
||||
- `media_api.rs:758-763` - no JPEG validation before serving
|
||||
|
||||
---
|
||||
|
||||
## Current Quality Mechanisms
|
||||
|
||||
### Endpoint 1: `face_thumbnail`
|
||||
|
||||
| Mechanism | Status | Location |
|
||||
|-----------|--------|----------|
|
||||
| Representative frame selection | Present | `tkg::query_auto_representative_frame()` |
|
||||
| ffmpeg success check | Present | `output.status.success()` |
|
||||
| JPEG validation | Missing | - |
|
||||
| Size validation | Missing | - |
|
||||
| Black frame detection | Missing | - |
|
||||
| Retry mechanism | Missing | - |
|
||||
|
||||
### Endpoint 2: `get_trace_thumbnail`
|
||||
|
||||
| Mechanism | Status | Location |
|
||||
|-----------|--------|----------|
|
||||
| Blur detection (candidate selection) | Present | `select_rep_face()` lines 463-480 |
|
||||
| Confidence filter (>0.7) | Present | `select_rep_face()` line 429 |
|
||||
| QC metadata filter | Present | `select_rep_face()` line 430 |
|
||||
| ffmpeg success check | Present | `status.status.success()` |
|
||||
| JPEG validation | Missing | - |
|
||||
| Black frame detection (extraction) | Missing | - |
|
||||
| Retry mechanism | Missing | - |
|
||||
|
||||
**Note**: `select_rep_face()` has sophisticated quality control for SELECTING the representative face, but the actual EXTRACTION step lacks validation.
|
||||
|
||||
---
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
### A. Input Data Problems
|
||||
|
||||
| Problem | Impact | Condition |
|
||||
|---------|--------|-----------|
|
||||
| `frame_number > total_frames` | Empty image | TKG returns wrong frame, user passes invalid value |
|
||||
| `crop exceeds dimensions` | Black frame / error | face bbox incorrect, video resolution changed |
|
||||
| Video file missing | 500 error | File deleted/moved |
|
||||
| Codec不支持seek | Empty/corrupted | Some codecs only support sequential read |
|
||||
|
||||
### B. ffmpeg Execution Problems
|
||||
|
||||
| Problem | Impact | Cause |
|
||||
|---------|--------|-------|
|
||||
| `select` no output | Empty JPEG | frame超出範圍 → ffmpeg skips all frames |
|
||||
| Pipe interrupted | Corrupted JPEG | stdout buffer full, ffmpeg terminated early |
|
||||
| `-ss` imprecise | Wrong frame | input seeking approximate, error ±5 frames |
|
||||
| crop failure | Black frame / 500 | `x+w > width` or `y+h > height` |
|
||||
|
||||
### C. Quality Control Gaps
|
||||
|
||||
| Gap | Impact | Current |
|
||||
|-----|--------|---------|
|
||||
| No JPEG validation | Corrupted image served | Only checks exit code |
|
||||
| No size check | 0 bytes returned | No output length check |
|
||||
| No black detection | Black frame served | blurdetect only in candidate selection |
|
||||
| No retry | Single failure = error | No retry mechanism |
|
||||
|
||||
---
|
||||
|
||||
## Concrete Failure Cases
|
||||
|
||||
### Case 1: Frame Exceeds Range
|
||||
|
||||
```
|
||||
Video: total_frames=1000 (DB record)
|
||||
Actual: video has only 950 frames (file truncated)
|
||||
Request: frame=980
|
||||
ffmpeg: select=eq(n\,980) → no match
|
||||
Output: 0 bytes JPEG
|
||||
Frontend: blank image
|
||||
```
|
||||
|
||||
### Case 2: Crop Exceeds Dimensions
|
||||
|
||||
```
|
||||
Video: 1920x1080
|
||||
face_bbox: x=1850, y=1050, w=100, h=100
|
||||
ffmpeg: crop=100:100:1850:1050
|
||||
Result: x+100=1950 > 1920 → ffmpeg error or black border
|
||||
```
|
||||
|
||||
### Case 3: Seek Imprecise
|
||||
|
||||
```
|
||||
Video: 25fps
|
||||
Request: frame=1000 (40 seconds)
|
||||
ffmpeg -ss 40.0 -i video
|
||||
Actual: seeks to frame 995~1005 range
|
||||
Result: extracts different face than select_rep_face chose
|
||||
```
|
||||
|
||||
### Case 4: Pipe Interrupted
|
||||
|
||||
```
|
||||
ffmpeg -i large_video -vf select=eq(n\,50000) -f image2pipe -
|
||||
Video large, select needs scan to frame 50000
|
||||
Pipe buffer full → ffmpeg may be killed or terminate early
|
||||
Output: incomplete JPEG (missing FFD9 footer)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Recommended Fixes
|
||||
|
||||
### Phase P0: Critical (Must Implement)
|
||||
|
||||
| Fix | Description | LOC | Location |
|
||||
|-----|-------------|-----|----------|
|
||||
| **Frame validation** | `frame <= total_frames` | ~20 | `media_api.rs:707-718` |
|
||||
| **Crop validation** | `x+w <= width, y+h <= height` | ~15 | `media_api.rs:731-735` |
|
||||
| **JPEG header check** | `data[0..3] == [0xFF,0xD8,0xFF]` | ~10 | Helper function |
|
||||
| **JPEG footer check** | `data[-2..] == [0xFF,0xD9]` | ~10 | Helper function |
|
||||
| **Minimum size check** | `data.len() > 100` | ~5 | Helper function |
|
||||
|
||||
### Phase P1: Important (Should Implement)
|
||||
|
||||
| Fix | Description | LOC | Location |
|
||||
|-----|-------------|-----|----------|
|
||||
| **Black frame detection** | ffmpeg `-vf blackdetect` filter | ~30 | After extraction |
|
||||
| **Output seeking** | Move `-ss` after `-i` for precision | ~5 | `trace_agent_api.rs:527` |
|
||||
|
||||
### Phase P2: Enhancement (Nice to Have)
|
||||
|
||||
| Fix | Description | LOC | Location |
|
||||
|-----|-------------|-----|----------|
|
||||
| **Retry mechanism** | Max 3 attempts, offset +30 frames each | ~50 | Both endpoints |
|
||||
| **Fallback frame** | Extract middle frame if all fail | ~30 | Both endpoints |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Step 1: Create Validation Module
|
||||
|
||||
Create `src/core/thumbnail/validator.rs`:
|
||||
|
||||
```rust
|
||||
pub fn validate_jpeg(data: &[u8]) -> Result<()> {
|
||||
// P0-1: Minimum size
|
||||
if data.len() < 100 {
|
||||
bail!("JPEG too small: {} bytes", data.len());
|
||||
}
|
||||
|
||||
// P0-2: JPEG header (SOI marker)
|
||||
if data[0..3] != [0xFF, 0xD8, 0xFF] {
|
||||
bail!("Invalid JPEG header");
|
||||
}
|
||||
|
||||
// P0-3: JPEG footer (EOI marker)
|
||||
if data[data.len()-2..] != [0xFF, 0xD9] {
|
||||
bail!("Incomplete JPEG");
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Add Frame/Crop Validation
|
||||
|
||||
In `media_api.rs`:
|
||||
|
||||
```rust
|
||||
// P0-4: Validate frame number
|
||||
let total_frames: i64 = sqlx::query_scalar(...)
|
||||
.bind(&file_uuid)
|
||||
.fetch_one(pool)
|
||||
.await?;
|
||||
|
||||
if frame > total_frames {
|
||||
return Err(StatusCode::BAD_REQUEST);
|
||||
}
|
||||
|
||||
// P0-5: Validate crop dimensions
|
||||
if let (Some(x), Some(y), Some(w), Some(h)) = (q.x, q.y, q.w, q.h) {
|
||||
let (width, height): (i32, i32) = sqlx::query_as(...)
|
||||
.bind(&file_uuid)
|
||||
.fetch_one(pool)
|
||||
.await?;
|
||||
|
||||
if x + w > width || y + h > height {
|
||||
return Err(StatusCode::BAD_REQUEST);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Integrate Validation
|
||||
|
||||
In both endpoints, after ffmpeg extraction:
|
||||
|
||||
```rust
|
||||
// Apply validation
|
||||
validate_jpeg(&output.stdout)
|
||||
.map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Test Cases
|
||||
|
||||
| Test | Input | Expected |
|
||||
|------|-------|----------|
|
||||
| Valid frame | `frame=500` (valid) | JPEG returned |
|
||||
| Frame exceeds | `frame=999999` | 400 BAD_REQUEST |
|
||||
| Valid crop | `x=100,y=100,w=200,h=200` | JPEG returned |
|
||||
| Crop exceeds | `x=1800,y=1000,w=200,h=200` | 400 BAD_REQUEST |
|
||||
| Empty video | corrupted video file | 500 INTERNAL_ERROR |
|
||||
| Black frame | fade-out frame | Retry or fallback |
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `src/core/thumbnail/mod.rs` | Add validator module |
|
||||
| `src/core/thumbnail/validator.rs` | New file (validation helpers) |
|
||||
| `src/api/media_api.rs` | Add validation in `face_thumbnail()` |
|
||||
| `src/api/trace_agent_api.rs` | Add validation in `get_trace_thumbnail()` |
|
||||
|
||||
---
|
||||
|
||||
## Estimated Effort
|
||||
|
||||
| Phase | LOC | Time |
|
||||
|-------|-----|------|
|
||||
| P0 (Critical) | ~60 | 1-2 days |
|
||||
| P1 (Important) | ~35 | 1 day |
|
||||
| P2 (Enhancement) | ~80 | 2-3 days |
|
||||
| **Total** | ~175 | 4-6 days |
|
||||
|
||||
---
|
||||
|
||||
## Version History
|
||||
|
||||
| Version | Date | Author | Changes |
|
||||
|---------|------|--------|---------|
|
||||
| 1.0.0 | 2026-05-27 | M5Max128 | Initial analysis complete |
|
||||
|
||||
---
|
||||
|
||||
## Next Steps for M5Max48
|
||||
|
||||
1. Read this document
|
||||
2. Implement P0 fixes first
|
||||
3. Test with edge cases
|
||||
4. Add P1/P2 as needed
|
||||
5. Update `AGENTS.md` if adding new validation commands
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- `docs_v1.0/DESIGN/Processor_Refactoring_Assessment.md` - Processor refactoring priorities
|
||||
- `src/api/media_api.rs:700-764` - face_thumbnail implementation
|
||||
- `src/api/trace_agent_api.rs:394-556` - select_rep_face and get_trace_thumbnail
|
||||
- `ffmpeg -vf blackdetect` documentation
|
||||
Reference in New Issue
Block a user