Add Charade face matching experience report
Documents the journey from Rust pipeline snowball bug through 5 iterations of pgvector-based matching to the final 11-identity centroid approach with dual-gate and ambiguity cleanup.
This commit is contained in:
Accusys
255
docs/CHARADE_FACE_MATCHING_EXPERIENCE.md
Normal file
255
docs/CHARADE_FACE_MATCHING_EXPERIENCE.md
Normal file
@@ -0,0 +1,255 @@
|
||||
# Charade 臉部匹配經驗總結
|
||||
|
||||
## 背景
|
||||
|
||||
Charade (1963) 影片 `a6fb22eebefaef17e62af874997c5944` 有 62,298 個人臉偵測結果，分布在 4,378 個 trace 中（TKG face tracker 輸出）。目標是將每張臉匹配到正確的 TMDb 演員 identity。
|
||||
|
||||
## 問題
|
||||
|
||||
### 1. Rust Pipeline (`face_agent.rs`) 的 Snowball 效應
|
||||
|
||||
原始 pipeline 透過多輪 propagation 來匹配：
|
||||
- Seed embedding 匹配 → propagation rounds (2-10 輪)
|
||||
- 每輪把已匹配的 face 當作新 seed 繼續擴散
|
||||
- 結果：**Antonio Passalia 被匹配 18,821 張臉**（實際應 < 50）
|
||||
- 原因：propagation 會放大初始匹配中的假陽性
|
||||
|
||||
### 2. Dev 資料庫污染
|
||||
|
||||
`dev` schema 的 `identity_bindings` 表：
|
||||
- 所有 trace-type binding 的 `file_uuid` 都是 NULL（12,828 行）
|
||||
- 這些 binding 只對應已刪除的 CCBN 檔案 (`63acd3bb`)
|
||||
- **完全無法用於 sync 到 public schema**
|
||||
|
||||
### 3. TMDb Seed Embedding 品質不均
|
||||
|
||||
22/23 個 TMDb identity 有 face_embedding（Thomas Chelimsky 因無 TMDb 照片而缺少）。但這些 seed 來自單一 TMDb 照片，品質差異大：
|
||||
|
||||
| Identity | Seed 品質 | 問題 |
|
||||
|----------|:---------:|:----:|
|
||||
| Audrey Hepburn | ✅ 高 | 特徵明顯，易區分 |
|
||||
| Cary Grant | ✅ 中 | 但 Charade 造型與 seed 照片有差異 |
|
||||
| Walter Matthau | ❌ 低 | Seed 照片與 Charade 形象差異大 |
|
||||
| Bernard Musson | ❌ 泛用 | 「典型白人男性」— seed 太泛用 |
|
||||
| Antonio Passalia | ❌ 泛用 | 同上 |
|
||||
|
||||
## 解決方案演進
|
||||
|
||||
### V1：直接 pgvector 比對 (threshold 0.50)
|
||||
|
||||
```sql
|
||||
CROSS JOIN LATERAL (
|
||||
SELECT i.id FROM identities i
|
||||
WHERE 1 - (embedding <=> i.face_embedding) >= 0.50
|
||||
ORDER BY 1 - (embedding <=> i.face_embedding) DESC LIMIT 1
|
||||
)
|
||||
```
|
||||
|
||||
**結果**：17,066 匹配 (27.4%)
|
||||
- ✅ Audrey 9,550 (正確)
|
||||
- ✅ Antonio 降為 151 (不再 snowball)
|
||||
- ❌ Bernard Musson 847／Paul Bonifas 273 — generic seed 假陽性
|
||||
- ❌ trace-level 衝突（同一 trace 多個 identity）
|
||||
- ❌ Walter Matthau 僅 535（seed 不準導致 recall 低）
|
||||
|
||||
### V2：Trace Conflict Cleanup
|
||||
|
||||
在 V1 之後，對每個 conflict trace 做多數決 → 清除 minority identity。
|
||||
|
||||
**結果**：移除 836 個污染臉
|
||||
- ✅ trace-level 衝突降為 0
|
||||
- ❌ Bernard Musson 仍保留 847（trace 內獨佔）
|
||||
- ❌ 無法解決 generic seed 的根本問題
|
||||
|
||||
### V3：雙階段 Centroid Matching
|
||||
|
||||
設計：
|
||||
|
||||
```
|
||||
Phase 1: Seed matching @ 0.55 (stricter) → 乾淨 base set
|
||||
Phase 2: Centroid matching @ 0.45 → 用電影內平均臉擴張 recall
|
||||
```
|
||||
|
||||
**結果**：27,375 匹配 (43.9%) → trace cleanup → 24,286 (39.0%)
|
||||
- ✅ Audrey 11,347 (+19%)
|
||||
- ✅ Cary Grant 3,107 (+56%)
|
||||
- ✅ Walter Matthau 1,200 (+124%) — centroid 修正 seed!
|
||||
- ❌ **Bernard Musson 2,903 (+243%)** — centroid 放大 generic seed
|
||||
- ❌ **Antonio Passalia 898 (+642%)** — 同上
|
||||
|
||||
**教訓**：Generic seed 的 centroid 更泛用。Phase 2 的低 threshold 讓問題惡化。
|
||||
|
||||
### V4：雙重驗證 (Dual Gate)
|
||||
|
||||
在 V3 的 Phase 2 加上 seed_sim >= 0.40 條件：
|
||||
|
||||
```
|
||||
centroid_sim >= 0.45 AND seed_sim >= 0.40
|
||||
```
|
||||
|
||||
**結果**：23,023 匹配 → gap cleanup → trace cleanup → **22,548 (36.2%)**
|
||||
- ✅ Bernard / Paul / Antonio / Michel / Clément / Raoul / Roger 仍偏高但 avg_seed_sim 改善
|
||||
|
||||
### V5（最終版）：排除 7 個 Generic Identity
|
||||
|
||||
核心洞察：**與其過濾假陽性，不如不讓 generic seed 參賽**。
|
||||
|
||||
只保留 11 個可靠的 TMDb identity，排除 7 個：
|
||||
- 排除：Bernard Musson · Paul Bonifas · Michel Thomass · Antonio Passalia · Clément Harari · Raoul Delfosse · Roger Trapp
|
||||
- 保留：Audrey · Cary · James Coburn · Jacques Marin · Walter Matthau · George Kennedy · Dominique Minot · Monte Landis · Stanley Donen · Ned Glass · Louis Viret
|
||||
|
||||
流程：
|
||||
|
||||
```
|
||||
1. Clear all assignments
|
||||
2. Phase 1 @ 0.55 — only against 11 identities
|
||||
3. Compute centroids
|
||||
4. Phase 2 — centroid>=0.45 AND seed>=0.40 (11 centroids)
|
||||
5. Ambiguity gate (top2 gap < 0.04 → NULL)
|
||||
6. Trace conflict cleanup
|
||||
```
|
||||
|
||||
**最終結果**：
|
||||
|
||||
| Identity | 最終 faces | traces | fpt | avg_sim |
|
||||
|----------|:----------:|:------:|:---:|:-------:|
|
||||
| Audrey Hepburn | 11,325 | 438 | 25.9 | 0.608 |
|
||||
| Cary Grant | **5,101** ≪ 大幅增加 | 269 | 19.0 | 0.497 |
|
||||
| James Coburn | 1,508 | 92 | 16.4 | 0.588 |
|
||||
| Jacques Marin | 1,438 | 84 | 17.1 | 0.631 |
|
||||
| Walter Matthau | 1,250 | 55 | 22.7 | 0.494 |
|
||||
| George Kennedy | 869 | 60 | 14.5 | 0.590 |
|
||||
| 排除的 7 個 | **0** ✅ | — | — | — |
|
||||
| Unassigned | 39,750 | — | — | — |
|
||||
|
||||
**Cary Grant 從 3,107→5,101 (+64%)**：之前被 Bernard/Antonio 攔截的臉全部釋放。
|
||||
|
||||
## 關鍵教訓
|
||||
|
||||
### 1. Generic Seed 辨識
|
||||
|
||||
可以透過以下指標辨識 generic seed：
|
||||
- **Phase 1 faces / traces 比例低**（< 5 fpt）
|
||||
- **被分配到大量短 trace**（表示非連續場景）
|
||||
- **avg_seed_sim 偏低但 face count 異常高**
|
||||
|
||||
### 2. Propagation 是雙面刃
|
||||
|
||||
Rust pipeline 的 propagation 可以增加 recall，但前提是 seed 要夠純。Generic seed + propagation = snowball。
|
||||
|
||||
### 3. Seed 數量 vs 品質
|
||||
|
||||
> 不是 identity 越多越好。11 個好 seed 勝過 22 個（含 7 個壞的）。
|
||||
|
||||
壞 seed 會攔截好 seed 的配對。排除壞 seed 後，那些臉自然會配到正確的人。
|
||||
|
||||
### 4. Centroid Matching 的適用條件
|
||||
|
||||
Centroid matching 只有在以下情況才有效：
|
||||
- Centroid 來自高信賴的 Phase 1 配對（threshold >= 0.55）
|
||||
- Centroid 的 Phase 1 base set > 200 faces
|
||||
- 搭配 seed_sim dual gate 防止 centroid 飄移
|
||||
|
||||
### 5. Trace Context 的重要性
|
||||
|
||||
- 一個 trace = 同一人（face tracker 保證）
|
||||
- Trace-level conflict cleanup 是必要的後處理
|
||||
- 但無法解決 trace 層級以下（同一 trace 內）的 contamination
|
||||
|
||||
## 可改進的方向
|
||||
|
||||
### 短期
|
||||
|
||||
1. **手動檢查 Cary Grant 的 5,101 faces**：avg_sim 0.497 偏低，部分可能是假陽性
|
||||
2. **補回已被排除的 identity**：對 Bernard Musson 等用更高 threshold（如 0.60 seed）只看能否 match 到少數高信賴臉
|
||||
3. **降低 Ambiguity Gate threshold**：從 0.04 降到 0.03 可再清除一批邊緣配對
|
||||
|
||||
### 中期
|
||||
|
||||
4. **多 seed 策略**：對每個 identity 用 3-5 張 TMDb 照片，取 centroid 作為 seed
|
||||
5. **場景約束**：利用 shot boundary 資訊限制跨場景的 identity 分配
|
||||
6. **雙向驗證**：同時用 face→identity 和 identity→trace 兩種方向互相驗證
|
||||
|
||||
### 長期
|
||||
|
||||
7. **取代 pgvector face-level matching**：改用 trace-level embedding（同一 trace 的所有 face 取平均），再對 trace 做 identity 匹配，減少 single-frame noise
|
||||
|
||||
## SQL 核心語法
|
||||
|
||||
### pgvector Nearest Neighbor
|
||||
|
||||
```sql
|
||||
SELECT fd.id, m.identity_id
|
||||
FROM eligible fd
|
||||
CROSS JOIN LATERAL (
|
||||
SELECT i.id FROM identities i
|
||||
WHERE 1 - (fd.embedding::vector <=> i.face_embedding) >= {threshold}
|
||||
ORDER BY 1 - (fd.embedding::vector <=> i.face_embedding) DESC
|
||||
LIMIT 1
|
||||
) m
|
||||
```
|
||||
|
||||
### Centroid 計算
|
||||
|
||||
```sql
|
||||
CREATE TABLE centroids AS
|
||||
SELECT identity_id, AVG(embedding::vector) as centroid
|
||||
FROM face_detections
|
||||
WHERE file_uuid = '{uuid}' AND identity_id IS NOT NULL
|
||||
GROUP BY identity_id
|
||||
HAVING COUNT(*) >= 5;
|
||||
```
|
||||
|
||||
### Trace Conflict Cleanup
|
||||
|
||||
```sql
|
||||
WITH conflict_traces AS (
|
||||
SELECT trace_id FROM face_detections
|
||||
WHERE file_uuid = '{uuid}' AND identity_id IS NOT NULL
|
||||
GROUP BY trace_id HAVING COUNT(DISTINCT identity_id) > 1
|
||||
),
|
||||
trace_majority AS (
|
||||
SELECT DISTINCT ON (ct.trace_id) ct.trace_id, fd.identity_id
|
||||
FROM conflict_traces ct
|
||||
JOIN face_detections fd ON fd.trace_id = ct.trace_id
|
||||
WHERE fd.file_uuid = '{uuid}' AND fd.identity_id IS NOT NULL
|
||||
GROUP BY ct.trace_id, fd.identity_id
|
||||
ORDER BY ct.trace_id, COUNT(*) DESC
|
||||
)
|
||||
UPDATE face_detections fd SET identity_id = NULL
|
||||
FROM trace_majority tm
|
||||
WHERE fd.file_uuid = '{uuid}' AND fd.trace_id = tm.trace_id
|
||||
AND fd.identity_id != tm.identity_id;
|
||||
```
|
||||
|
||||
### Ambiguity Gate
|
||||
|
||||
```sql
|
||||
WITH all_sims AS (
|
||||
SELECT fd.id, c.identity_id,
|
||||
1 - (fd.embedding::vector <=> c.centroid) as sim
|
||||
FROM face_detections fd
|
||||
CROSS JOIN centroids c
|
||||
WHERE fd.file_uuid = '{uuid}' AND fd.identity_id IS NOT NULL
|
||||
),
|
||||
ranked AS (
|
||||
SELECT id, sim, LEAD(sim) OVER (PARTITION BY id ORDER BY sim DESC) as sim2
|
||||
FROM all_sims
|
||||
),
|
||||
ambiguous AS (
|
||||
SELECT id FROM ranked
|
||||
WHERE rn = 1 AND sim - COALESCE(sim2, 0) < 0.04
|
||||
)
|
||||
UPDATE face_detections fd SET identity_id = NULL
|
||||
FROM ambiguous a WHERE fd.id = a.id;
|
||||
```
|
||||
|
||||
## 資料庫備份
|
||||
|
||||
每次關鍵操作都有備份：
|
||||
|
||||
| Backup | Rows | 內容 |
|
||||
|--------|:----:|:------|
|
||||
| `fd_charade_bak` | 62,298 | 原始無 identity 的 Charade face_detections |
|
||||
| `fd_state_bak2` | 24,286 | V5 執行前的 assignment snapshot |
|
||||
| `wp_snippets_backup_20260601_11940.sql` | — | WordPress snippets 備份 |
|
||||
Reference in New Issue
Block a user