docs: Phase 3 object identity design (v3 model)
- Object instance tracking (similar to face trace) - Custom detector for stamps, guns, etc. - TKG integration for object-face-speaker graph - Upgrade path: yolov5nu → yolov8m, fine-tune, zero-shot
This commit is contained in:
@@ -28,7 +28,7 @@ Search / Query / Identity APIs
|
||||
|-----------|------------------|------|---------|
|
||||
| `{uuid}_v1` | `momentry_dev_v1` | sentence chunk embedding(base) | ASR + ASRX + Rule 1 完成 |
|
||||
| `{uuid}_v2` | `momentry_dev_v2` | 完整 pipeline + 5W1H | 全部完成 |
|
||||
| `{uuid}_v3` | - | 預留後續升級 | - |
|
||||
| `{uuid}_v3` | `momentry_dev_v3` | object identity + custom detector | v2 + object instance matching 完成 |
|
||||
|
||||
各版本共存不覆蓋。
|
||||
|
||||
@@ -110,5 +110,56 @@ release/
|
||||
| 類比 | 訓練好的 weights | inference engine |
|
||||
| 內容 | `.json` + chunks + vectors | Rust binary |
|
||||
| 生命週期 | 每部影片產出一個 | 一個 binary 服務所有影片 |
|
||||
| 版本 | `{uuid}_v1`(base) / `{uuid}_v2` | `momentry_playground` / `momentry` |
|
||||
| 版本 | `{uuid}_v1`(base) / `{uuid}_v2` / `{uuid}_v3` | `momentry_playground` / `momentry` |
|
||||
| 交付對象 | 終端使用者 | 部署工程師 |
|
||||
|
||||
---
|
||||
|
||||
## Phase 3:Object Identity(v3 model)
|
||||
|
||||
### 目標
|
||||
|
||||
從影片中提取關鍵物體(郵票、手槍、信封、放大鏡...),對同類物體做 instance-level 的跨畫面追蹤與辨識,達到類似 face trace 的效果 — 不只是 detect class,還能區分「這一張郵票」vs「那一張郵票」。
|
||||
|
||||
### 現狀問題
|
||||
|
||||
1. **COCO 80 類不包含關鍵物體** — 郵票、手槍、信封、放大鏡等不在 COCO 資料集中
|
||||
2. **YOLOv5nano 偵測率低** — 即使是 COCO 類別(knife, cell phone)在 nano 模型上 recall 不足
|
||||
3. **無 object instance matching** — 目前只有 frame-level detection,沒有跨 frame 的物體追蹤
|
||||
|
||||
### 技術方向
|
||||
|
||||
```
|
||||
YOLOv8m/OWL-ViT → 改善 detection coverage
|
||||
↓
|
||||
Object Tracker (IoU + embedding,類似 face tracker)
|
||||
↓
|
||||
object_trace → TKG CO_OCCURS_WITH edges
|
||||
↓
|
||||
object identity → 同物體跨場景辨識
|
||||
```
|
||||
|
||||
| 方向 | 方法 | 效果 |
|
||||
|------|------|------|
|
||||
| Model upgrade | `yolov5nu` → `yolov8s.pt` / `yolov8m.pt` | COCO recall 提升 |
|
||||
| Custom fine-tune | 收集 stamps/guns 資料 fine-tune YOLO | 可偵測非 COCO 物件 |
|
||||
| Zero-shot | OWL-ViT / Grounding DINO by text prompt | 不用 training,但速度慢 |
|
||||
| Object trace | IoU + embedding 跨 frame 匹配 | instance-level 追蹤 |
|
||||
| Object identity | clustering 跨場景辨識同一物體 | 可在全片搜尋「這把槍」 |
|
||||
|
||||
### 與 TKG 整合
|
||||
|
||||
```
|
||||
face_trace -[:CO_OCCURS_WITH]-> object_instance:5 (這把槍)
|
||||
face_trace -[:CO_OCCURS_WITH]-> object_instance:42 (這張郵票)
|
||||
|
||||
查詢: "Audrey Hepburn 拿這把槍的畫面"
|
||||
→ face_trace:5 -[:SPEAKS_AS]-> SPEAKER_0
|
||||
→ face_trace:5 -[:CO_OCCURS_WITH]-> object_instance:5
|
||||
```
|
||||
|
||||
### 交付順序
|
||||
|
||||
1. YOLO model upgrade(低難度,立即見效)
|
||||
2. Object tracker(中難度,參考 face tracker 實作)
|
||||
3. Custom fine-tune / zero-shot(高難度,需資料或新模型)
|
||||
|
||||
Reference in New Issue
Block a user