feat: TKG extension - pose data + mutual gaze detection

This commit is contained in:
Accusys
2026-05-22 07:09:54 +08:00
parent a9e9285032
commit deb9516796
3 changed files with 393 additions and 49 deletions

View File

@@ -415,7 +415,96 @@ ORDER BY cooccurrence_count DESC
---
## 6. Phase 計畫
## 7. TKG 擴充Pose + Mutual Gaze
### 7.1 現狀
| 元件 | 有 pose | 有 mutual gaze |
|------|-----------|-----------------|
| `face.json` | ✅ yaw/pitch/roll | ❌ 未計算 |
| `face_detections.metadata` | ❌ 無 | ❌ 無 |
| `tkg_nodes` (face_trace) | ❌ 無 | — |
| `tkg_edges` (CO_OCCURS_WITH) | ❌ 無 | ❌ 無 |
### 7.2 目標
```
face_processor → face.json (有 pose)
tkg.rs (TKG builder)
├── 讀取 face.json 的 pose 資料
├── 算 avg_yaw/pitch/roll → 寫入 face_trace node
├── 對同框 face_trace 配對,判斷 mutual_gaze
└── 寫入 CO_OCCURS_WITH edge properties
tkg_edges.properties.mutual_gaze = true
API 查詢 → 互看 + 面積最大 → 代表 frame
```
### 7.3 face_trace node 新增 properties
```json
{
"frame_count": 53,
"start_frame": 38165,
"end_frame": 38321,
"avg_bbox": {"x": 731, "y": 215, "width": 228, "height": 228},
"avg_yaw": 0.014,
"avg_pitch": 0.224,
"avg_roll": -0.069
}
```
### 7.4 CO_OCCURS_WITH edge 新增 properties
```json
{
"first_frame": 38187,
"frame_count": 156,
"mutual_gaze": true,
"yaw_a_avg": 0.021,
"yaw_b_avg": -0.421,
"gaze_angle_delta": 0.442
}
```
### 7.5 Mutual Gaze 判斷邏輯
```python
GAZE_THRESHOLD = 0.05 # rad
def detect_mutual_gaze(frame_a, frame_b):
# 判斷 A 和 B 的左右位置關係
if bbox_a.cx < bbox_b.cx:
# A 在左B 在右 → A 要看右B 要看左
return yaw_a > GAZE_THRESHOLD and yaw_b < -GAZE_THRESHOLD
else:
# A 在右B 在左 → A 要看左B 要看右
return yaw_a < -GAZE_THRESHOLD and yaw_b > GAZE_THRESHOLD
```
### 7.6 代表 Frame 選取邏輯(應用端)
```
1. 查 TKG: MATCH (a)-[e:CO_OCCURS_WITH]->(b) WHERE e.mutual_gaze = true
→ 回傳互看 frame_count 最高的 face_trace 配對
2. 取該配對的 frame 範圍內,面積×信心最高 frame
3. FFmpeg blurdetect → 選最清晰的作為代表
```
### 7.7 Timeline
| Phase | 內容 | 工時 |
|-------|------|------|
| 1 | tkg.rs 讀取 face.json 的 pose + 寫入 node | 4-6h |
| 2 | mutual_gaze 判斷 + 寫入 edge | 3-4h |
| 3 | 對 Charade 重跑 TKG + 驗證 | 1h |
| 4 | 代表 frame 選取邏輯 | 2-3h |
---
## 8. Phase 計畫
| Phase | Query Types | 預計工時 | 依賴 |
|-------|------------|---------|------|