admin/momentry_core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add Thumbnail QA Analysis for M5Max48 implementation

Browse Source
This commit is contained in:
M5Max128
2026-05-27 14:35:53 +08:00
parent c85794292a
commit a036d985b7
1 changed files with 340 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

340
docs_v1.0/DESIGN/Thumbnail_QA_Analysis.md Normal file
View File

@@ -0,0 +1,340 @@
---
title: Thumbnail Endpoint Quality Assurance Analysis
version: 1.0.0
date: 2026-05-27
author: M5Max128
status: research_complete
---
# Thumbnail Endpoint Quality Assurance Analysis
## Scope
| Item | Status |
|------|--------|
| Research | Complete |
| Implementation | Pending (M5Max48) |
| Affected Endpoints | 2 |
## Overview
Thumbnail endpoints currently lack quality validation, resulting in potential anomalies:
- **Empty images** - ffmpeg produces 0 bytes output
- **Black frames** - extracted frame is all black
- **Corrupted JPEG** - incomplete ffmpeg output
## Affected Endpoints
| Endpoint | File | Line |
|----------|------|------|
| `/api/v1/file/:file_uuid/thumbnail` | `src/api/media_api.rs` | 700-764 |
| `/api/v1/file/:file_uuid/trace/:trace_id/thumbnail` | `src/api/trace_agent_api.rs` | 514-556 |
---
## Anomaly Classification
### Type 1: Empty Image (No Frame)
**Symptom**: Returns 0 bytes or very small JPEG
**Root Causes**:
1. `frame_number > total_frames` - requested frame exceeds video length
2. Video file missing or corrupted
3. Codec does not support frame-level seek
4. ffmpeg `-vf select` filter finds no matching frame
**Code Locations**:
- `media_api.rs:710-716` - `query_auto_representative_frame()` may return invalid frame
- `media_api.rs:720-728` - `file_path` query may return non-existent file
- `media_api.rs:754-756` - only checks `output.status.success()`, not output content
### Type 2: Black Frame
**Symptom**: Returns valid JPEG but all black or very dark
**Root Causes**:
1. `crop` parameters exceed video dimensions (`x+w > width` or `y+h > height`)
2. Extracted frame is from fade-in/fade-out transition
3. Video has black opening/closing credits
4. Low-light scene
**Code Locations**:
- `media_api.rs:731-735` - crop validation missing
- `trace_agent_api.rs:530` - crop may exceed dimensions
### Type 3: Corrupted JPEG
**Symptom**: Returns incomplete JPEG (browser shows broken image)
**Root Causes**:
1. ffmpeg stdout pipe interrupted before completion
2. ffmpeg process killed mid-output
3. JPEG encoder failure
4. Incomplete write to stdout buffer
**Code Locations**:
- `media_api.rs:751` - pipe output may be truncated
- `media_api.rs:758-763` - no JPEG validation before serving
---
## Current Quality Mechanisms
### Endpoint 1: `face_thumbnail`
| Mechanism | Status | Location |
|-----------|--------|----------|
| Representative frame selection | Present | `tkg::query_auto_representative_frame()` |
| ffmpeg success check | Present | `output.status.success()` |
| JPEG validation | Missing | - |
| Size validation | Missing | - |
| Black frame detection | Missing | - |
| Retry mechanism | Missing | - |
### Endpoint 2: `get_trace_thumbnail`
| Mechanism | Status | Location |
|-----------|--------|----------|
| Blur detection (candidate selection) | Present | `select_rep_face()` lines 463-480 |
| Confidence filter (>0.7) | Present | `select_rep_face()` line 429 |
| QC metadata filter | Present | `select_rep_face()` line 430 |
| ffmpeg success check | Present | `status.status.success()` |
| JPEG validation | Missing | - |
| Black frame detection (extraction) | Missing | - |
| Retry mechanism | Missing | - |
**Note**: `select_rep_face()` has sophisticated quality control for SELECTING the representative face, but the actual EXTRACTION step lacks validation.
---
## Root Cause Analysis
### A. Input Data Problems
| Problem | Impact | Condition |
|---------|--------|-----------|
| `frame_number > total_frames` | Empty image | TKG returns wrong frame, user passes invalid value |
| `crop exceeds dimensions` | Black frame / error | face bbox incorrect, video resolution changed |
| Video file missing | 500 error | File deleted/moved |
| Codec不支持seek | Empty/corrupted | Some codecs only support sequential read |
### B. ffmpeg Execution Problems
| Problem | Impact | Cause |
|---------|--------|-------|
| `select` no output | Empty JPEG | frame超出範圍 → ffmpeg skips all frames |
| Pipe interrupted | Corrupted JPEG | stdout buffer full, ffmpeg terminated early |
| `-ss` imprecise | Wrong frame | input seeking approximate, error ±5 frames |
| crop failure | Black frame / 500 | `x+w > width` or `y+h > height` |
### C. Quality Control Gaps
| Gap | Impact | Current |
|-----|--------|---------|
| No JPEG validation | Corrupted image served | Only checks exit code |
| No size check | 0 bytes returned | No output length check |
| No black detection | Black frame served | blurdetect only in candidate selection |
| No retry | Single failure = error | No retry mechanism |
---
## Concrete Failure Cases
### Case 1: Frame Exceeds Range
```
Video: total_frames=1000 (DB record)
Actual: video has only 950 frames (file truncated)
Request: frame=980
ffmpeg: select=eq(n\,980) → no match
Output: 0 bytes JPEG
Frontend: blank image
```
### Case 2: Crop Exceeds Dimensions
```
Video: 1920x1080
face_bbox: x=1850, y=1050, w=100, h=100
ffmpeg: crop=100:100:1850:1050
Result: x+100=1950 > 1920 → ffmpeg error or black border
```
### Case 3: Seek Imprecise
```
Video: 25fps
Request: frame=1000 (40 seconds)
ffmpeg -ss 40.0 -i video
Actual: seeks to frame 995~1005 range
Result: extracts different face than select_rep_face chose
```
### Case 4: Pipe Interrupted
```
ffmpeg -i large_video -vf select=eq(n\,50000) -f image2pipe -
Video large, select needs scan to frame 50000
Pipe buffer full → ffmpeg may be killed or terminate early
Output: incomplete JPEG (missing FFD9 footer)
```
---
## Recommended Fixes
### Phase P0: Critical (Must Implement)
| Fix | Description | LOC | Location |
|-----|-------------|-----|----------|
| **Frame validation** | `frame <= total_frames` | ~20 | `media_api.rs:707-718` |
| **Crop validation** | `x+w <= width, y+h <= height` | ~15 | `media_api.rs:731-735` |
| **JPEG header check** | `data[0..3] == [0xFF,0xD8,0xFF]` | ~10 | Helper function |
| **JPEG footer check** | `data[-2..] == [0xFF,0xD9]` | ~10 | Helper function |
| **Minimum size check** | `data.len() > 100` | ~5 | Helper function |
### Phase P1: Important (Should Implement)
| Fix | Description | LOC | Location |
|-----|-------------|-----|----------|
| **Black frame detection** | ffmpeg `-vf blackdetect` filter | ~30 | After extraction |
| **Output seeking** | Move `-ss` after `-i` for precision | ~5 | `trace_agent_api.rs:527` |
### Phase P2: Enhancement (Nice to Have)
| Fix | Description | LOC | Location |
|-----|-------------|-----|----------|
| **Retry mechanism** | Max 3 attempts, offset +30 frames each | ~50 | Both endpoints |
| **Fallback frame** | Extract middle frame if all fail | ~30 | Both endpoints |
---
## Implementation Plan
### Step 1: Create Validation Module
Create `src/core/thumbnail/validator.rs`:
```rust
pub fn validate_jpeg(data: &[u8]) -> Result<()> {
// P0-1: Minimum size
if data.len() < 100 {
bail!("JPEG too small: {} bytes", data.len());
}
// P0-2: JPEG header (SOI marker)
if data[0..3] != [0xFF, 0xD8, 0xFF] {
bail!("Invalid JPEG header");
}
// P0-3: JPEG footer (EOI marker)
if data[data.len()-2..] != [0xFF, 0xD9] {
bail!("Incomplete JPEG");
}
Ok(())
}
```
### Step 2: Add Frame/Crop Validation
In `media_api.rs`:
```rust
// P0-4: Validate frame number
let total_frames: i64 = sqlx::query_scalar(...)
.bind(&file_uuid)
.fetch_one(pool)
.await?;
if frame > total_frames {
return Err(StatusCode::BAD_REQUEST);
}
// P0-5: Validate crop dimensions
if let (Some(x), Some(y), Some(w), Some(h)) = (q.x, q.y, q.w, q.h) {
let (width, height): (i32, i32) = sqlx::query_as(...)
.bind(&file_uuid)
.fetch_one(pool)
.await?;
if x + w > width || y + h > height {
return Err(StatusCode::BAD_REQUEST);
}
}
```
### Step 3: Integrate Validation
In both endpoints, after ffmpeg extraction:
```rust
// Apply validation
validate_jpeg(&output.stdout)
.map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;
```
---
## Testing Strategy
### Test Cases
| Test | Input | Expected |
|------|-------|----------|
| Valid frame | `frame=500` (valid) | JPEG returned |
| Frame exceeds | `frame=999999` | 400 BAD_REQUEST |
| Valid crop | `x=100,y=100,w=200,h=200` | JPEG returned |
| Crop exceeds | `x=1800,y=1000,w=200,h=200` | 400 BAD_REQUEST |
| Empty video | corrupted video file | 500 INTERNAL_ERROR |
| Black frame | fade-out frame | Retry or fallback |
---
## Files to Modify
| File | Changes |
|------|---------|
| `src/core/thumbnail/mod.rs` | Add validator module |
| `src/core/thumbnail/validator.rs` | New file (validation helpers) |
| `src/api/media_api.rs` | Add validation in `face_thumbnail()` |
| `src/api/trace_agent_api.rs` | Add validation in `get_trace_thumbnail()` |
---
## Estimated Effort
| Phase | LOC | Time |
|-------|-----|------|
| P0 (Critical) | ~60 | 1-2 days |
| P1 (Important) | ~35 | 1 day |
| P2 (Enhancement) | ~80 | 2-3 days |
| **Total** | ~175 | 4-6 days |
---
## Version History
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0.0 | 2026-05-27 | M5Max128 | Initial analysis complete |
---
## Next Steps for M5Max48
1. Read this document
2. Implement P0 fixes first
3. Test with edge cases
4. Add P1/P2 as needed
5. Update `AGENTS.md` if adding new validation commands
---
## References
- `docs_v1.0/DESIGN/Processor_Refactoring_Assessment.md` - Processor refactoring priorities
- `src/api/media_api.rs:700-764` - face_thumbnail implementation
- `src/api/trace_agent_api.rs:394-556` - select_rep_face and get_trace_thumbnail
- `ffmpeg -vf blackdetect` documentation