feat: TKG extension - pose data + mutual gaze detection
This commit is contained in:
@@ -415,7 +415,96 @@ ORDER BY cooccurrence_count DESC
|
||||
|
||||
---
|
||||
|
||||
## 6. Phase 計畫
|
||||
## 7. TKG 擴充:Pose + Mutual Gaze
|
||||
|
||||
### 7.1 現狀
|
||||
|
||||
| 元件 | 有 pose? | 有 mutual gaze? |
|
||||
|------|-----------|-----------------|
|
||||
| `face.json` | ✅ yaw/pitch/roll | ❌ 未計算 |
|
||||
| `face_detections.metadata` | ❌ 無 | ❌ 無 |
|
||||
| `tkg_nodes` (face_trace) | ❌ 無 | — |
|
||||
| `tkg_edges` (CO_OCCURS_WITH) | ❌ 無 | ❌ 無 |
|
||||
|
||||
### 7.2 目標
|
||||
|
||||
```
|
||||
face_processor → face.json (有 pose)
|
||||
↓
|
||||
tkg.rs (TKG builder)
|
||||
├── 讀取 face.json 的 pose 資料
|
||||
├── 算 avg_yaw/pitch/roll → 寫入 face_trace node
|
||||
├── 對同框 face_trace 配對,判斷 mutual_gaze
|
||||
└── 寫入 CO_OCCURS_WITH edge properties
|
||||
↓
|
||||
tkg_edges.properties.mutual_gaze = true
|
||||
↓
|
||||
API 查詢 → 互看 + 面積最大 → 代表 frame
|
||||
```
|
||||
|
||||
### 7.3 face_trace node 新增 properties
|
||||
|
||||
```json
|
||||
{
|
||||
"frame_count": 53,
|
||||
"start_frame": 38165,
|
||||
"end_frame": 38321,
|
||||
"avg_bbox": {"x": 731, "y": 215, "width": 228, "height": 228},
|
||||
"avg_yaw": 0.014,
|
||||
"avg_pitch": 0.224,
|
||||
"avg_roll": -0.069
|
||||
}
|
||||
```
|
||||
|
||||
### 7.4 CO_OCCURS_WITH edge 新增 properties
|
||||
|
||||
```json
|
||||
{
|
||||
"first_frame": 38187,
|
||||
"frame_count": 156,
|
||||
"mutual_gaze": true,
|
||||
"yaw_a_avg": 0.021,
|
||||
"yaw_b_avg": -0.421,
|
||||
"gaze_angle_delta": 0.442
|
||||
}
|
||||
```
|
||||
|
||||
### 7.5 Mutual Gaze 判斷邏輯
|
||||
|
||||
```python
|
||||
GAZE_THRESHOLD = 0.05 # rad
|
||||
|
||||
def detect_mutual_gaze(frame_a, frame_b):
|
||||
# 判斷 A 和 B 的左右位置關係
|
||||
if bbox_a.cx < bbox_b.cx:
|
||||
# A 在左,B 在右 → A 要看右,B 要看左
|
||||
return yaw_a > GAZE_THRESHOLD and yaw_b < -GAZE_THRESHOLD
|
||||
else:
|
||||
# A 在右,B 在左 → A 要看左,B 要看右
|
||||
return yaw_a < -GAZE_THRESHOLD and yaw_b > GAZE_THRESHOLD
|
||||
```
|
||||
|
||||
### 7.6 代表 Frame 選取邏輯(應用端)
|
||||
|
||||
```
|
||||
1. 查 TKG: MATCH (a)-[e:CO_OCCURS_WITH]->(b) WHERE e.mutual_gaze = true
|
||||
→ 回傳互看 frame_count 最高的 face_trace 配對
|
||||
2. 取該配對的 frame 範圍內,面積×信心最高 frame
|
||||
3. FFmpeg blurdetect → 選最清晰的作為代表
|
||||
```
|
||||
|
||||
### 7.7 Timeline
|
||||
|
||||
| Phase | 內容 | 工時 |
|
||||
|-------|------|------|
|
||||
| 1 | tkg.rs 讀取 face.json 的 pose + 寫入 node | 4-6h |
|
||||
| 2 | mutual_gaze 判斷 + 寫入 edge | 3-4h |
|
||||
| 3 | 對 Charade 重跑 TKG + 驗證 | 1h |
|
||||
| 4 | 代表 frame 選取邏輯 | 2-3h |
|
||||
|
||||
---
|
||||
|
||||
## 8. Phase 計畫
|
||||
|
||||
| Phase | Query Types | 預計工時 | 依賴 |
|
||||
|-------|------------|---------|------|
|
||||
|
||||
Reference in New Issue
Block a user