feat: backup architecture docs, source code, and scripts
This commit is contained in:
563
docs_v1.0/AI_AGENTS/CONTEXT/METADATA_PROCESSORS.md
Normal file
563
docs_v1.0/AI_AGENTS/CONTEXT/METADATA_PROCESSORS.md
Normal file
@@ -0,0 +1,563 @@
|
||||
# Momentry Core - Metadata 及 處理器總覽
|
||||
|
||||
本文檔說明 Momentry Core 中 chunks 資料表的 metadata 結構,以及各類處理器的輸出欄位。
|
||||
|
||||
## 1. Chunks 資料表結構
|
||||
|
||||
### 1.1 直接欄位 (Direct Columns)
|
||||
|
||||
這些欄位直接儲存於 chunks 資料表中:
|
||||
|
||||
| 欄位 | 類型 | 來源處理器 | 說明 |
|
||||
|------|------|----------|------|
|
||||
| `id` | serial | 系統 | 主鍵 |
|
||||
| `uuid` | varchar(32) | 系統 | 影片 UUID |
|
||||
| `chunk_id` | varchar(64) | 系統 | Chunk ID (如 sentence_0001) |
|
||||
| `chunk_index` | integer | 系統 | 順序編號 |
|
||||
| `chunk_type` | varchar(32) | 系統 | sentence/cut/time |
|
||||
| `text_content` | text | ASR processor | 語音轉文字結果 |
|
||||
| `content` | jsonb | - | 原始內容 (rule, data 等) |
|
||||
| `metadata` | jsonb | 多個處理器 | 參閱下方 1.2 |
|
||||
| `visual_stats` | jsonb | add_yolo_to_chunks.py | YOLO 識別結果 |
|
||||
| `speaker_ids` | text[] | ASRX processor | 說話者 ID 陣列 |
|
||||
| `face_ids` | integer[] | Face processor | 臉部 ID 陣列 |
|
||||
| `summary_text` | text | generate_chunk_summaries.py | LLM 生成摘要 |
|
||||
| `parent_chunk_id` | varchar(64) | 系統 | 父 chunk ID |
|
||||
| `fps` | double | ffprobe | 幀率 |
|
||||
| `start_frame` | bigint | ffprobe | 開始幀 |
|
||||
| `end_frame` | bigint | ffprobe | 結束幀 |
|
||||
| `metadata_version` | integer | 系統 | Metadata 版本 (5W1H, identity, visual) |
|
||||
| `content_version` | integer | 系統 | Content 版本 (text_content, summary_text) |
|
||||
| `created_at` | timestamp | 系統 | 建立時間 |
|
||||
| `updated_at` | timestamp | 系統 | 最後更新時間 |
|
||||
|
||||
### 版本控制說明
|
||||
|
||||
| 欄位 | 說明 | 遞增時機 |
|
||||
|------|------|----------|
|
||||
| `metadata_version` | Metadata 版本 | 更新 5W1H, identity, visual 時 |
|
||||
| `content_version` | Content 版本 | 更新 text_content, summary_text 時 |
|
||||
| `updated_at` | 最後更新時間 | 任何更新時自動更新 |
|
||||
|
||||
**判別更新語法**:
|
||||
|
||||
```sql
|
||||
-- 檢查哪些 chunk 需要重新生成 5W1H
|
||||
SELECT chunk_id, metadata_version, content_version, updated_at
|
||||
FROM dev.chunks
|
||||
WHERE metadata_version < 1;
|
||||
|
||||
-- 檢查特定時間後的更新
|
||||
SELECT chunk_id, updated_at
|
||||
FROM dev.chunks
|
||||
WHERE updated_at > '2024-01-01';
|
||||
|
||||
-- 檢查版本差異 (需要重新處理)
|
||||
SELECT c.*
|
||||
FROM dev.chunks c
|
||||
WHERE c.metadata_version <
|
||||
(SELECT MAX(metadata_version) FROM dev.chunks WHERE uuid = c.uuid);
|
||||
```
|
||||
|
||||
## 11. 動態 Metadata 管理
|
||||
|
||||
### 11.1 欄位動態增減
|
||||
|
||||
Metadata JSONB 支援動態欄位,可根據處理器執行結果動態添加:
|
||||
|
||||
```python
|
||||
# 動態添加欄位
|
||||
metadata = existing_metadata or {}
|
||||
metadata[field_name] = value
|
||||
UPDATE chunks SET metadata = metadata || %s::jsonb
|
||||
```
|
||||
|
||||
### 11.2 常見動態欄位
|
||||
|
||||
| 欄位 | 新增時機 | 來源處理器 |
|
||||
|------|----------|------------|
|
||||
| `chunk_5w1h` | 生成 summary | generate_chunk_summaries.py |
|
||||
| `chunk_identity` | ASRX/Face 執行後 | 來源欄位聚合 |
|
||||
| `chunk_visual` | YOLO 執行後 | add_yolo_to_chunks.py |
|
||||
| `chunk_emotion` | 情緒分析 | future emotion_processor.py |
|
||||
| `chunk_pose` | 姿勢辨識 | future pose_processor.py |
|
||||
| `chunk_sentiment` | 情感分析 | future sentiment_processor.py |
|
||||
|
||||
### 11.3 版本升級策略
|
||||
|
||||
每次重大更新時遞增版本號:
|
||||
|
||||
```python
|
||||
if新增重大欄位:
|
||||
metadata_version += 1
|
||||
# 記錄變更日誌
|
||||
```
|
||||
|
||||
### 11.4 重跑機制
|
||||
|
||||
```bash
|
||||
# 重跑特定版本後的 chunk
|
||||
python scripts/generate_chunk_summaries.py --uuid <uuid> --min-version 1
|
||||
|
||||
# 查看版本分佈
|
||||
SELECT metadata_version, COUNT(*)
|
||||
FROM dev.chunks
|
||||
GROUP BY metadata_version;
|
||||
```
|
||||
|
||||
### 1.2 Metadata 結構 (JSONB)
|
||||
|
||||
`metadata` 欄位包含多個子欄位,由不同處理器產生:
|
||||
|
||||
```json
|
||||
{
|
||||
"chunk_5w1h": {
|
||||
"who": "演員或角色",
|
||||
"what": "主要動作或事件",
|
||||
"when": "時間上下文",
|
||||
"where": "地點",
|
||||
"why": "目的或原因",
|
||||
"how": "表達方式"
|
||||
},
|
||||
"chunk_identity": {
|
||||
"speakers": ["speaker_001", "speaker_002"],
|
||||
"faces": ["face_1", "face_3"]
|
||||
},
|
||||
"chunk_visual": {
|
||||
"objects": ["person", "car", "tree"],
|
||||
"places": ["street", "office"]
|
||||
},
|
||||
"structured_summary": {
|
||||
"who": "Parent 級別角色",
|
||||
"what": "Parent 級別動作",
|
||||
...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| 子欄位 | 類型 | 來源處理器 | 說明 |
|
||||
|--------|------|----------|------|
|
||||
| `chunk_5w1h` | jsonb | generate_chunk_summaries.py | Chunk 級別的 5W1H + Emotion + Actions |
|
||||
| `chunk_5w1h.who` | string | person | 人物名稱 (含來源標記) |
|
||||
| `chunk_5w1h.what` | string | action | 具體動作 |
|
||||
| `chunk_5w1h.when` | string | position | 場景中位置 (beginning/middle/end) |
|
||||
| `chunk_5w1h.where` | string | location | 地點 |
|
||||
| `chunk_5w1h.why` | string | purpose | 目的 |
|
||||
| `chunk_5w1h.how` | string | manner | 表達方式 |
|
||||
| `chunk_5w1h.emotion` | string | emotion | 情緒/語氣 |
|
||||
| `chunk_5w1h.actions` | string[] | verbs | 動作動詞 |
|
||||
| `chunk_identity` | jsonb | 來源欄位聚合 | speaker_ids + face_ids 資訊 |
|
||||
| `chunk_visual` | jsonb | add_yolo_to_chunks.py | YOLO 物體識別結果 |
|
||||
| `structured_summary` | jsonb | regenerate_parent_5w1h.py | Parent 級別 5W1H + tone + characters + key_events |
|
||||
|
||||
### chunk_5w1h 欄位說明 (Chunk 級)
|
||||
|
||||
| 欄位 | 類型 | 說明 | 範例 |
|
||||
|------|------|------|------|
|
||||
| `who` | string | 此 chunk 出現的角色 (含來源) | "John (SPEAKER_1), Mary (face_3)" |
|
||||
| `what` | string | 此 chunk 的具體動作 | "Giving warning" |
|
||||
| `when` | string | 相對時間位置 | "Mid-scene" |
|
||||
| `where` | string | 地點 (如提及) | "Near taxi" |
|
||||
| `why` | string | 此動作的目的 | "Warn about danger" |
|
||||
| `how` | string | 表達/呈現方式 | "Urgent tone" |
|
||||
| `emotion` | string | 情緒/語氣 | "Fearful, urgent" |
|
||||
| `actions` | string[] | 動作動詞 | ["run", "shout", "warn"] |
|
||||
|
||||
**Prompt 增強內容**:
|
||||
- 從 person_identities 取得驗證的人物名稱
|
||||
- 包含 speaker_id 和 face_id 來源標記
|
||||
- 視覺辨識: objects, places, actions
|
||||
- Time range 傳入 chunk 時間範圍
|
||||
- Emotion + Actions 額外欄位
|
||||
|
||||
### chunk_identity 欄位說明
|
||||
|
||||
| 欄位 | 類型 | 說明 | 範例 |
|
||||
|------|------|------|------|
|
||||
| `speakers` | string[] | 說話者 ID | ["speaker_001", "speaker_002"] |
|
||||
| `faces` | string[] | 臉部 ID | ["face_1", "face_3"] |
|
||||
| `global_identity` | string | 對應的全局人物 ID | "person_001" |
|
||||
| `person_name` | string | 識別的人物名稱 | "John" |
|
||||
|
||||
> 說明:
|
||||
> - `speakers`/`faces` 來自 ASRX/Face processor
|
||||
> - `global_identity` 來自 `person_identities` 表,關聯 face_identity_id
|
||||
> - `person_name` 來自 `person_identities.name`,經過確認的人物名稱
|
||||
|
||||
### 全域人物 Identity (person_identities 表)
|
||||
|
||||
每個影片會識別並記錄出現的人物,儲存於 `dev.person_identities` 表:
|
||||
|
||||
| 欄位 | 類型 | 說明 |
|
||||
|------|------|------|
|
||||
| `person_id` | varchar(255) | 人物唯一 ID (如 person_001) |
|
||||
| `name` | varchar(255) | 人物名稱 (可確認) |
|
||||
| `speaker_id` | varchar(255) | 對應的說話者 ID |
|
||||
| `video_uuid` | varchar(255) | 影片 UUID |
|
||||
| `face_identity_id` | integer | 對應的 global identity |
|
||||
| `appearance_count` | integer | 出現次數 |
|
||||
| `first_appearance_time` | double | 首次出現時間 |
|
||||
| `last_appearance_time` | double | 最後出現時間 |
|
||||
| `confidence` | double | 辨識信心度 |
|
||||
| `is_confirmed` | boolean | 是否已確認 |
|
||||
|
||||
### 全域 Identity (face_identities 表)
|
||||
|
||||
跨影片的全局人物身份:
|
||||
|
||||
| 欄位 | 類型 | 說明 |
|
||||
|------|------|------|
|
||||
| `id` | serial | 主鍵 |
|
||||
| `face_id` | integer | 臉部 ID |
|
||||
| `name` | varchar(255) | 識別姓名 |
|
||||
| `embedding` | blob | 人臉向量特徵 |
|
||||
|
||||
### 人物識別流程
|
||||
|
||||
Momentry 的人物識別分為三個層級:
|
||||
|
||||
```
|
||||
層級 1: 原始識別 (chunks 表)
|
||||
├── chunks.face_ids → 臉部 ID (local to chunk)
|
||||
└── chunks.speaker_ids → 說話者 ID (local to chunk)
|
||||
|
||||
層級 2: 影片級識別 (person_identities 表)
|
||||
├── person_id → 人物 ID (影片內唯一)
|
||||
├── name → 識別出的人物名稱 (如 "John")
|
||||
├── speaker_id → 對應的說話者
|
||||
└── face_identity_id → 對應的全局 Identity
|
||||
|
||||
層級 3: 全局身份 (face_identities 表)
|
||||
├── id → 全局唯一 ID
|
||||
├── face_id → 臉部特徵 ID
|
||||
├── name → 確認的姓名
|
||||
└── embedding → 人臉向量 (用於比對)
|
||||
```
|
||||
|
||||
**識別流程說明**:
|
||||
|
||||
```
|
||||
Step 1: ASRX Processor
|
||||
chunks.speaker_ids ← 說話者分離
|
||||
|
||||
Step 2: Face Processor
|
||||
chunks.face_ids ← 臉部偵測
|
||||
|
||||
Step 3: Auto-identify
|
||||
person_identities ← 合併 speaker + face (影片級)
|
||||
|
||||
Step 4: Global Matching
|
||||
face_identities ← 人臉向量比對 (全局 Identity)
|
||||
↑
|
||||
合併相同人臉者為同一 Identity
|
||||
```
|
||||
|
||||
**命名原則**:
|
||||
|
||||
- `person_id` = 角色名 (如 "John", "Adam")
|
||||
- 而非 "Person_8"
|
||||
- 透過 speaker 對應 + 手動確認
|
||||
|
||||
**範例**:
|
||||
|
||||
```sql
|
||||
-- 取得影片中的人物列表
|
||||
SELECT person_id, name, speaker_id, appearance_count
|
||||
FROM dev.person_identities
|
||||
WHERE video_uuid = '384b0ff44aaaa1f1'
|
||||
ORDER BY appearance_count DESC;
|
||||
|
||||
-- 取得 chunk 的人物
|
||||
SELECT c.chunk_id, pi.name, pi.speaker_id
|
||||
FROM dev.chunks c
|
||||
JOIN dev.person_identities pi ON c.uuid = pi.video_uuid
|
||||
WHERE c.chunk_id = 'sentence_0001';
|
||||
```
|
||||
|
||||
### 取得 chunk 的人物資訊
|
||||
|
||||
```sql
|
||||
-- 取得某 chunk 的人物
|
||||
SELECT pi.name, pi.speaker_id, pi.appearance_count
|
||||
FROM dev.person_identities pi
|
||||
JOIN dev.chunks c ON c.uuid = pi.video_uuid
|
||||
WHERE c.chunk_id = 'sentence_0001';
|
||||
```
|
||||
|
||||
### chunk_visual 欄位說明
|
||||
|
||||
| 欄位 | 類型 | 說明 | 範例 |
|
||||
|------|------|------|------|
|
||||
| `objects` | string[] | YOLO 識別物體 | ["person", "car", "tree"] |
|
||||
| `places` | string[] | Places365 識別地點 | ["street", "office"] |
|
||||
|
||||
## 2. 處理器對照表
|
||||
|
||||
### 2.1 ASR 處理器 (語音辨識)
|
||||
|
||||
**用途**:將影片音軌轉換為文字
|
||||
|
||||
| 處理器 | 輸出欄位 | 說明 |
|
||||
|--------|---------|------|
|
||||
| asr_processor_small_multilingual.py | text_content | Small 模型,多語言 |
|
||||
| asr_processor_simplified.py | text_content | 簡化版 |
|
||||
| asr_processor_contract_v1.py | text_content | 契約版本 v1 |
|
||||
| asr_processor_contract_v2.py | text_content | 契約版本 v2 |
|
||||
|
||||
**輸出**:
|
||||
- `text_content`: 語音轉文字結果
|
||||
- 寫入 `chunks.content` 和 `chunks.text_content`
|
||||
|
||||
### 2.2 ASRX 處理器 (增強說話者辨識)
|
||||
|
||||
**用途**:說話者分離 (Diarization)
|
||||
|
||||
| 處理器 | 輸出欄位 | 說明 |
|
||||
|--------|---------|------|
|
||||
| asrx_processor.py | speaker_ids | 標準版 |
|
||||
| asrx_processor_contract_v1.py | speaker_ids | 契約版 v1 |
|
||||
|
||||
**輸出**:
|
||||
- `speaker_ids`: 說話者 ID 陣列,如 `["speaker_001", "speaker_002"]`
|
||||
- 目前為空 `{}`,需執行後才會填充
|
||||
|
||||
### 2.3 Face 處理器 (臉部偵測)
|
||||
|
||||
**用途**:偵測並追蹤人臉
|
||||
|
||||
| 處理器 | 輸出欄位 | 說明 |
|
||||
|--------|---------|------|
|
||||
| analyze_video_faces.py | face_ids | 臉部偵測 |
|
||||
|
||||
**輸出**:
|
||||
- `face_ids`: 臉部 ID 陣列,如 `[1, 3, 5]`
|
||||
- 目前為空 `{}`,需執行後才會填充
|
||||
|
||||
### 2.4 YOLO 處理器 (物體識別)
|
||||
|
||||
**用途**:識別場景中的物體和地點
|
||||
|
||||
| 處理器 | 輸出欄位 | 說明 |
|
||||
|--------|---------|------|
|
||||
| add_yolo_to_chunks.py | visual_stats, chunk_visual | YOLO + Places365 |
|
||||
|
||||
**輸出**:
|
||||
- `visual_stats`: 原始識別結果
|
||||
- `metadata.chunk_visual`: 簡化格式 `{objects: [...], places: [...]}`
|
||||
|
||||
### 2.5 Summary 處理器 (生成摘要)
|
||||
|
||||
**用途**:生成 chunk 摘要和 5W1H 分析
|
||||
|
||||
| 處理器 | 輸出欄位 | 說明 |
|
||||
|--------|---------|------|
|
||||
| generate_chunk_summaries.py | summary_text, chunk_5w1h, chunk_identity, chunk_visual | LLM 生成 |
|
||||
| regenerate_parent_5w1h.py | structured_summary | Parent 場景級 5W1H |
|
||||
|
||||
**輸入**:
|
||||
- chunk.text_content
|
||||
- parent_chunks.summary_text
|
||||
- parent_chunks.metadata.structured_summary
|
||||
- chunk.speaker_ids (用於 chunk_identity)
|
||||
- chunk.face_ids (用於 chunk_identity)
|
||||
- chunk.visual_stats (用於 chunk_visual)
|
||||
|
||||
**輸出**:
|
||||
- `summary_text`: 2-3 句摘要
|
||||
- `metadata.chunk_5w1h`: Who/What/When/Where/Why/How
|
||||
- `metadata.chunk_identity`: speakers, faces
|
||||
- `metadata.chunk_visual`: objects, places
|
||||
|
||||
## 3. Parent Chunks 結構
|
||||
|
||||
Parent chunks 代表場景 (scene) 層級:
|
||||
|
||||
| 欄位 | 類型 | 說明 |
|
||||
|------|------|------|
|
||||
| `id` | serial | 主鍵 |
|
||||
| `uuid` | varchar(32) | 影片 UUID |
|
||||
| `scene_order` | integer | 場景順序 |
|
||||
| `summary_text` | text | 場景摘要 (LLM 生成) |
|
||||
| `metadata` | jsonb | 包含 structured_summary |
|
||||
|
||||
### Parent Metadata 結構
|
||||
|
||||
```json
|
||||
{
|
||||
"structured_summary": {
|
||||
"who": "主要角色",
|
||||
"what": "主要事件",
|
||||
"when": "時間線",
|
||||
"where": "地點",
|
||||
"why": "動機",
|
||||
"how": "方式",
|
||||
"tone": ["緊張", "懸疑", "溫馨"],
|
||||
"characters": ["角色A", "角色B", "角色C"],
|
||||
"key_events": ["事件1", "事件2", "事件3"],
|
||||
"summary_5lines": "5行摘要..."
|
||||
},
|
||||
"auto_generated_by": "gemma4",
|
||||
"chunk_count": 885
|
||||
}
|
||||
```
|
||||
|
||||
### structured_summary 欄位說明
|
||||
|
||||
| 欄位 | 類型 | 說明 | 範例 |
|
||||
|------|------|------|------|
|
||||
| `who` | string | 主要角色 | "Mr. Balletman, Adam" |
|
||||
| `what` | string | 主要動作或事件 | "Escape attempt" |
|
||||
| `when` | string | 時間上下文 | "During critical moment" |
|
||||
| `where` | string | 地點 | "Near taxi" |
|
||||
| `why` | string | 動機或原因 | "Evade capture" |
|
||||
| `how` | string | 執行方式 | "Quickly moving to taxi" |
|
||||
| `tone` | string[] | 語氣/情緒 | ["Urgent", "Tense", "Fearful"] |
|
||||
| `characters` | string[] | 場景中的角色 | ["Mr. Balletman", "Adam", "Antagonist"] |
|
||||
| `key_events` | string[] | 關鍵事件 | ["Decision to flee", "Warning given"] |
|
||||
| `summary_5lines` | string | 5行摘要 | "Line 1\nLine 2..." |
|
||||
|
||||
## 4. Chunk 類型說明
|
||||
|
||||
| 類型 | 需要搜尋 | 說明 |
|
||||
|------|----------|------|
|
||||
| `sentence` | ✓ | 有 text_content,需向量化存入 Qdrant |
|
||||
| `cut` | ✗ | 場景剪輯點,無文字內容 |
|
||||
| `time` | ✗ | 時間區間標記,無文字 |
|
||||
|
||||
**搜尋適用性**:
|
||||
- sentence: 有文字內容,可進行語意搜尋
|
||||
- cut/time: 無文字,僅供時間定位使用
|
||||
|
||||
## 5. 處理流程 (Pipeline)
|
||||
|
||||
```
|
||||
1. ffprobe → 取得影片資訊 (fps, frame count)
|
||||
2. ASR processor → text_content
|
||||
3. [ASRX processor] → speaker_ids (選用)
|
||||
4. [Face processor] → face_ids (選用)
|
||||
5. add_yolo_to_chunks.py → visual_stats
|
||||
6. generate_chunk_summaries.py → summary_text + metadata
|
||||
7. [vectorize_chunk_summaries.py] → Qdrant 向量
|
||||
```
|
||||
|
||||
## 6. Qdrant Collections
|
||||
|
||||
| Collection | 向量類型 | 用途 |
|
||||
|------------|----------|------|
|
||||
| `momentry_dev_chunk_summaries` | nomic-embed-text | Chunk summary 語意搜尋 |
|
||||
| `momentry_dev_vectors` | 原始向量 | 備用 |
|
||||
|
||||
## 7. API 回傳格式
|
||||
|
||||
Chunk Detail API 合併 chunk 和 parent 的 metadata:
|
||||
|
||||
```
|
||||
metadata
|
||||
├── chunk_5w1h (chunk 級)
|
||||
├── chunk_identity (chunk 級)
|
||||
├── chunk_visual (chunk 級)
|
||||
├── structured_summary (parent 級) ← 只在有 parent 時
|
||||
├── auto_generated_by
|
||||
└── chunk_count
|
||||
```
|
||||
|
||||
## 8. 執行狀態檢查
|
||||
|
||||
```bash
|
||||
# 檢查 summary 生成進度
|
||||
psql -h localhost -U accusys -d momentry -c "
|
||||
SELECT COUNT(*) as total,
|
||||
COUNT(CASE WHEN summary_text IS NOT NULL THEN 1 END) as generated
|
||||
FROM dev.chunks WHERE chunk_type = 'sentence';"
|
||||
|
||||
# 檢查執行中的處理器
|
||||
ps aux | grep -E "processor|generate" | grep -v grep
|
||||
|
||||
# 檢查 visual_stats
|
||||
psql -h localhost -U accusys -d momentry -c "
|
||||
SELECT COUNT(*) FROM dev.chunks WHERE visual_stats IS NOT NULL;"
|
||||
```
|
||||
|
||||
## 9. 待執行處理器
|
||||
|
||||
### 人物識別處理器 (依序執行)
|
||||
|
||||
```bash
|
||||
# Step 1: ASRX 執行說話者分離
|
||||
python scripts/asrx_processor.py --uuid 384b0ff44aaaa1f1
|
||||
|
||||
# Step 2: Face 執行臉部偵測
|
||||
python scripts/analyze_video_faces.py --uuid 384b0ff44aaaa1f1
|
||||
|
||||
# Step 3: Auto-identify 建立影片級人物
|
||||
python scripts/auto_identify_persons.py --uuid 384b0ff44aaaa1f1
|
||||
|
||||
# Step 4: 全局 Identity 比對 (需累積一定數量的 face_identities)
|
||||
python scripts/match_faces_to_identities.py
|
||||
|
||||
# Step 5: 重新生成 chunk 5W1H (包含新的 identity 資訊)
|
||||
python scripts/generate_chunk_summaries.py --uuid 384b0ff44aaaa1f1
|
||||
```
|
||||
|
||||
### 檢查待處理狀態
|
||||
|
||||
```bash
|
||||
# 檢查 speaker_ids
|
||||
psql -h localhost -U accusys -d momentry -c "
|
||||
SELECT COUNT(*) FROM dev.chunks
|
||||
WHERE speaker_ids IS NOT NULL AND array_length(speaker_ids, 1) > 0;"
|
||||
|
||||
# 檢查 face_ids
|
||||
psql -h localhost -U accusys -d momentry -c "
|
||||
SELECT COUNT(*) FROM dev.chunks
|
||||
WHERE face_ids IS NOT NULL AND array_length(face_ids, 1) > 0;"
|
||||
|
||||
# 檢查 person_identities
|
||||
psql -h localhost -U accusys -d momentry -c "
|
||||
SELECT COUNT(*) FROM dev.person_identities
|
||||
WHERE video_uuid = '384b0ff44aaaa1f1';"
|
||||
|
||||
# 檢查 face_identities (全局)
|
||||
psql -h localhost -U accusys -d momentry -c "
|
||||
SELECT COUNT(*) FROM dev.face_identities;"
|
||||
```
|
||||
|
||||
## 10. 自動化重新生成機制
|
||||
|
||||
### 觸發條件
|
||||
|
||||
當以下事件發生時,應自動重新生成 chunk 的 5W1H 和相關 metadata:
|
||||
|
||||
| 事件 | 觸發動作 |
|
||||
|------|----------|
|
||||
| 第一次執行 ASRX | 重新生成含 speaker_ids 的 5W1H |
|
||||
| 第一次執行 Face | 重新生成含 face_ids 的 5W1H |
|
||||
| 新增 chunk | 為新 chunk 生成 5W1H |
|
||||
| 修改 chunk 內容 | 更新 5W1H 和 summary |
|
||||
| 新增/修改 speaker | 重新生成含新 speaker 的 5W1H |
|
||||
| 新增/修改 face | 重新生成含新 face 的 5W1H |
|
||||
|
||||
### 重新生成流程
|
||||
|
||||
```
|
||||
事件觸發
|
||||
↓
|
||||
更新 speaker_ids / face_ids / person_identities
|
||||
↓
|
||||
呼叫 generate_chunk_summaries.py --uuid <uuid> --regenerate
|
||||
↓
|
||||
重新產生:
|
||||
├── summary_text (2-3 句)
|
||||
├── metadata.chunk_5w1h (Who/What/When/Where/Why/How)
|
||||
├── metadata.chunk_identity (更新後的 speakers/faces)
|
||||
└── metadata.chunk_visual (若 visual_stats 有更新)
|
||||
```
|
||||
|
||||
### 重點
|
||||
|
||||
每次處理器執行後,Chunk metadata 會包含最新的:
|
||||
1. **speaker_ids** → 進入 `chunk_identity.speakers`
|
||||
2. **face_ids** → 進入 `chunk_identity.faces`
|
||||
3. **person_identities** → 進入 `chunk_identity.person_name`
|
||||
|
||||
確保 LLM 產生的 5W1H 包含最新的角色資訊。
|
||||
116
docs_v1.0/AI_AGENTS/CORE/AGENT_SPEC.md
Normal file
116
docs_v1.0/AI_AGENTS/CORE/AGENT_SPEC.md
Normal file
@@ -0,0 +1,116 @@
|
||||
# AI Agent 設計規範 (Agent Design Specification)
|
||||
|
||||
| 項目 | 內容 |
|
||||
|------|------|
|
||||
| 建立者 | OpenCode |
|
||||
| 建立時間 | 2026-04-25 |
|
||||
| 文件版本 | V1.0 |
|
||||
|
||||
---
|
||||
|
||||
## 版本歷史
|
||||
|
||||
| 版本 | 日期 | 目的 | 操作人 | 工具/模型 |
|
||||
|------|------|------|--------|-----------|
|
||||
| V1.0 | 2026-04-25 | 定義 Momentry Core 中 AI Agent 的標準設計與職責 | OpenCode | OpenCode |
|
||||
|
||||
---
|
||||
|
||||
## 1. 核心概念
|
||||
|
||||
在 Momentry Core 系統中,處理邏輯分為三個層次,本規範專注於第三層:
|
||||
|
||||
| 層次 | 名稱 | 特性 | 範例 |
|
||||
|------|------|------|------|
|
||||
| **L1** | **Processor (處理器)** | **確定性 (Deterministic)**<br>輸入 A 必得輸出 B。通常為編譯型程式或腳本。 | FFmpeg, Whisper (ASR), YOLO |
|
||||
| **L2** | **Rule (規則)** | **邏輯性 (Logic)**<br>基於明確的條件、正則表達式或時間軸聚合。 | 語句切分,時間重疊計算 |
|
||||
| **L3** | **Agent (智能體)** | **推論性 (Probabilistic)**<br>依賴 LLM 進行語義理解、決策或生成。具備 Prompt 或 Workflow。 | 5W1H 推論,身份解析,摘要生成 |
|
||||
|
||||
---
|
||||
|
||||
## 2. Agent 職責 (Responsibilities)
|
||||
|
||||
AI Agent 負責處理那些傳統程式難以精確定義規則的任務。
|
||||
**注意**: 在系統架構中,Agent 被視為一種 **資源 (Resource)**,與 Processor 和 Service 統一由 **資源註冊中心 (Resource Registry)** 管理。
|
||||
|
||||
1. **語義理解 (Semantic Understanding)**: 將非結構化數據(如 OCR 文字、雜訊 ASR 文本)轉化為結構化標籤 (5W1H)。
|
||||
2. **跨模態匹配 (Cross-Modal Matching)**: 綜合視覺、聽覺和文本證據,判斷「畫面中的臉」是否為「資料庫中的人」。
|
||||
3. **內容生成 (Content Generation)**: 為影片片段生成自然的摘要或標題。
|
||||
4. **查詢解析 (Query Parsing)**: 將用戶的自然語言請求轉譯為系統可執行的 API 調用序列。
|
||||
|
||||
---
|
||||
|
||||
## 3. 標準設計結構 (Design Structure)
|
||||
|
||||
所有 AI Agent 的設計文件必須遵循以下結構:
|
||||
|
||||
### 3.1 檔案命名
|
||||
* **格式**: `[AGENT_TYPE]_[PURPOSE].md`
|
||||
* **範例**: `CONTEXT_5W1H_INFERENCE.md`
|
||||
|
||||
### 3.2 文件內容
|
||||
|
||||
#### 3.2.1 Agent 目標 (Goal)
|
||||
簡短描述此 Agent 解決的業務問題。
|
||||
> **範例**: 從雜亂的 YOLO 標籤和 OCR 文本中推論場景的「地點」和「天氣」資訊。
|
||||
|
||||
#### 3.2.2 輸入數據 (Input)
|
||||
定義 Agent 接收的數據格式。通常來自 Processor 輸出或 Rule 產物。
|
||||
* **來源**: `PROCESSORS/` 或 `CHUNKING/`
|
||||
* **格式**: JSON, Text, List of Frames.
|
||||
|
||||
#### 3.2.3 核心邏輯 (Core Logic: Prompt / Workflow)
|
||||
這是 Agent 的靈魂。
|
||||
* **單一 Prompt Agent**: 提供完整的 System Prompt。
|
||||
```markdown
|
||||
## System Prompt
|
||||
You are a scene analysis assistant...
|
||||
```
|
||||
* **多步 Workflow Agent**: 提供步驟圖或偽代碼。
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Start] --> B[Extract Entities]
|
||||
B --> C[Verify with Knowledge Base]
|
||||
C --> D[Output Result]
|
||||
```
|
||||
|
||||
#### 3.2.4 輸出格式 (Output)
|
||||
定義 Agent 產出的結構化數據 (通常為 JSON)。
|
||||
```json
|
||||
{
|
||||
"who": ["Actor Name"],
|
||||
"what": ["Action"],
|
||||
"confidence": 0.95
|
||||
}
|
||||
```
|
||||
|
||||
#### 3.2.5 模型配置 (Model Config)
|
||||
建議使用的模型類型及其原因。
|
||||
* **推理模型 (Reasoning)**: `o1`, `R1` (用於複雜邏輯判斷)
|
||||
* **生成模型 (Generation)**: `GPT-4o`, `Sonnet` (用於摘要)
|
||||
* **本地模型 (Local)**: `Llama-3`, `Qwen` (用於隱私數據)
|
||||
|
||||
---
|
||||
|
||||
## 4. 開發工作流 (Development Workflow)
|
||||
|
||||
1. **定義需求**: 確定是否需要 AI 介入 (若規則可解,優先使用 Rule)。
|
||||
2. **撰寫 Prompt**: 在文檔中迭代 Prompt,直到達到穩定輸出。
|
||||
3. **工具串接**: 若需要外部數據 (如 TMDB),定義 Tool 定義。
|
||||
4. **實作封裝**: 將 Prompt/Workflow 封裝為 Rust/Python 模組,透過 API 調用。
|
||||
|
||||
---
|
||||
|
||||
## 5. 相關文件
|
||||
|
||||
* `UNIFIED_RESOURCE_REGISTRY.md` - 系統統一資源管理架構 (Agents 作為資源註冊)。
|
||||
* `AI_DRIVEN_PROCESSOR_CONTRACT.md` - Processor 層級的整合合約。
|
||||
* `CHUNKING_ARCHITECTURE.md` - Rule 層級的架構。
|
||||
* `FILE_IDENTITY_API_DESIGN.md` - 全局架構。
|
||||
|
||||
---
|
||||
|
||||
## 版本資訊
|
||||
|
||||
- 版本: V1.0
|
||||
- 建立日期: 2026-04-25
|
||||
248
docs_v1.0/AI_AGENTS/IDENTITY/FACE_SPEAKER_PERSON_API_GUIDE.md
Normal file
248
docs_v1.0/AI_AGENTS/IDENTITY/FACE_SPEAKER_PERSON_API_GUIDE.md
Normal file
@@ -0,0 +1,248 @@
|
||||
# Momentry Face / Speaker / Person API 開發指南
|
||||
|
||||
> **版本**: 3.5 | **更新日期**: 2026-04-17
|
||||
> **適用對象**: n8n 自動化流程開發者、Portal 前端開發者
|
||||
|
||||
---
|
||||
|
||||
## 快速開始
|
||||
|
||||
### 環境
|
||||
|
||||
| 環境 | URL | 說明 |
|
||||
|------|-----|------|
|
||||
| **正式版** | `https://api.momentry.ddns.net` | 外部存取 (HTTPS/TLSv1.3) |
|
||||
| **本機版** | `http://localhost:3002` | 同一台機器使用 (延遲更低) |
|
||||
|
||||
### 認證
|
||||
|
||||
所有 API 請求需在 Header 加入 API Key:
|
||||
|
||||
```bash
|
||||
curl https://api.momentry.ddns.net/api/v1/person/list \
|
||||
-H "X-API-Key: YOUR_API_KEY"
|
||||
```
|
||||
|
||||
**API Key**(marcom 團隊使用):
|
||||
```
|
||||
muser_68600856036340bcafc01930eb4bd839
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 鐵律:所有 Face/Speaker/Person API 都必須提供 video_uuid
|
||||
|
||||
**沒有例外。** 所有端點都需要 `video_uuid`。
|
||||
|
||||
```
|
||||
錯誤: GET /api/v1/person/list → 400 missing field `video_uuid`
|
||||
錯誤: GET /api/v1/person/Person_0 → 400 missing field `video_uuid`
|
||||
正確: GET /api/v1/person/list?video_uuid=xxx → 200 OK
|
||||
```
|
||||
|
||||
| 識別碼 | 全域唯一 | 說明 |
|
||||
|--------|:---:|------|
|
||||
| `chunk_id` | ❌ | 每部影片重新編號 |
|
||||
| `person_id` | ❌ | 每部影片有自己的 Person_0, Person_1... |
|
||||
| `speaker_id` | ❌ | 每部影片有自己的 SPEAKER_0, SPEAKER_1... |
|
||||
| **`video_uuid + person_id`** | ✅ | 唯一組合 |
|
||||
| **`video_uuid + chunk_id`** | ✅ | 唯一組合 |
|
||||
| `face_id` | ✅ | UUID 格式,全域唯一 |
|
||||
| `merge_id` | ✅ | UUID 格式,全域唯一 |
|
||||
|
||||
---
|
||||
|
||||
## API 端點總覽(全部需要 video_uuid)
|
||||
|
||||
| 端點 | 方法 | video_uuid 位置 | 說明 |
|
||||
|------|:---:|:---:|------|
|
||||
| `/api/v1/person/list` | GET | query | 列出人物 |
|
||||
| `/api/v1/person/auto-identify` | POST | body | 自動識別人 |
|
||||
| `/api/v1/person/suggest` | POST | body | AI 建議 |
|
||||
| `/api/v1/person/:id` | GET | query | 人物詳情 |
|
||||
| `/api/v1/person/:id` | PATCH | query | 更新人物 |
|
||||
| `/api/v1/person/:id/thumbnail` | GET | query | 臉部截圖 |
|
||||
| `/api/v1/person/:id/timeline` | GET | query | 出場時間軸 |
|
||||
| `/api/v1/person/:id/similar` | GET | query | 相似人物 |
|
||||
| `/api/v1/person/:id/appearances` | GET | query | 出場紀錄 |
|
||||
| `/api/v1/person/:id/unbind-speaker` | POST | body | 解除 Speaker |
|
||||
| `/api/v1/person/:id/reassign-speaker` | POST | body | 重新綁定 Speaker |
|
||||
| `/api/v1/person/:id/remove-appearance` | POST | body | 刪除出場紀錄 |
|
||||
| `/api/v1/person/:id/reassign-appearance` | POST | body | 轉移出場紀錄 |
|
||||
| `/api/v1/person/:id/split` | POST | body | 分割人物 |
|
||||
| `/api/v1/person/merge` | POST | body | 合併人物 |
|
||||
| `/api/v1/person/merge/undo` | POST | body | 撤銷合併 |
|
||||
| `/api/v1/person/merge/history` | GET | query | 合併歷史 |
|
||||
| `/api/v1/search/universal` | POST | body | 統一搜尋 |
|
||||
| `/api/v1/search/persons` | GET | query | 搜尋人物 |
|
||||
| `/api/v1/chunks/:id/persons` | GET | query | chunk 內人物 |
|
||||
| `/api/v1/face/register` | POST | body | 註冊臉孔 |
|
||||
| `/api/v1/face/list` | GET | query | 已註冊臉孔列表 |
|
||||
|
||||
---
|
||||
|
||||
## 詳細 API 說明
|
||||
|
||||
### 1. GET /api/v1/person/list
|
||||
|
||||
列出指定影片的人物。
|
||||
|
||||
**Query Parameters:**
|
||||
|
||||
| 參數 | 類型 | 必填 | 說明 |
|
||||
|------|:---:|:---:|------|
|
||||
| `video_uuid` | string | **是** | 影片 UUID |
|
||||
| `limit` | int | 否 | 每頁筆數 (預設 50) |
|
||||
| `offset` | int | 否 | 偏移量 (預設 0) |
|
||||
| `min_appearances` | int | 否 | 最低出場次數 |
|
||||
| `has_speaker` | bool | 否 | 僅顯示有 Speaker 的人物 |
|
||||
|
||||
**Request:**
|
||||
```
|
||||
GET /api/v1/person/list?video_uuid=384b0ff44aaaa1f1&limit=10&min_appearances=100
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"persons": [
|
||||
{
|
||||
"person_id": "Person_0",
|
||||
"name": null,
|
||||
"speaker_id": "SPEAKER_0",
|
||||
"appearance_count": 17832,
|
||||
"total_appearance_duration": 3600.5,
|
||||
"first_appearance_time": 79.56,
|
||||
"last_appearance_time": 6863.34,
|
||||
"is_confirmed": false,
|
||||
"speaker_confidence": 0.504
|
||||
}
|
||||
],
|
||||
"total": 303
|
||||
}
|
||||
```
|
||||
|
||||
### 2. GET /api/v1/person/:id
|
||||
|
||||
取得人物詳情。
|
||||
|
||||
**Query Parameters:**
|
||||
|
||||
| 參數 | 類型 | 必填 |
|
||||
|------|:---:|:---:|
|
||||
| `video_uuid` | string | **是** |
|
||||
|
||||
### 3. POST /api/v1/person/merge
|
||||
|
||||
合併多個人物為一人。
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"video_uuid": "384b0ff44aaaa1f1",
|
||||
"target_person_id": "Person_0",
|
||||
"source_person_ids": ["Person_4", "Person_25"]
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Merged 2 persons into Person_0",
|
||||
"target_person_id": "Person_0",
|
||||
"merge_id": "5b12e3ac-12fa-45c0-88e1-5cff67604a7d"
|
||||
}
|
||||
```
|
||||
|
||||
> ⚠️ **請儲存 `merge_id`**,以便日後撤銷合併。
|
||||
|
||||
### 4. POST /api/v1/search/universal
|
||||
|
||||
統一搜尋。
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"query": "stamp",
|
||||
"uuid": "384b0ff44aaaa1f1",
|
||||
"types": ["chunk", "person"],
|
||||
"limit": 20
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 影片定位:Frame 為主
|
||||
|
||||
**重要**: 所有影片位置都以 **frame (幀號)** 為唯一準確單位,time 僅供參考。
|
||||
|
||||
```json
|
||||
{
|
||||
"start_frame": 29795,
|
||||
"end_frame": 29963,
|
||||
"fps": 59.94,
|
||||
"start_time": 497.08,
|
||||
"end_time": 499.88
|
||||
}
|
||||
```
|
||||
|
||||
**轉換公式**: `time = frame / fps`
|
||||
|
||||
> ⚠️ **注意**: 所有搜尋 API (`/api/v1/search`, `/api/v1/n8n/search`, `/api/v1/search/universal`) 現在都統一回傳 `start_frame`, `end_frame`, `fps` 欄位,確保前端可以精確定位影片幀號。
|
||||
|
||||
---
|
||||
|
||||
## n8n 工作流範例
|
||||
|
||||
```
|
||||
[Webhook: video_processed]
|
||||
body: { "uuid": "384b0ff44aaaa1f1" }
|
||||
↓
|
||||
[HTTP: POST /api/v1/person/auto-identify]
|
||||
body: { "video_uuid": "{{ $json.uuid }}" }
|
||||
↓
|
||||
[HTTP: POST /api/v1/person/suggest]
|
||||
body: { "video_uuid": "{{ $json.uuid }}" }
|
||||
↓
|
||||
[IF: confidence >= 0.7]
|
||||
├─ YES → [HTTP: PATCH /api/v1/person/{{person_id}}?video_uuid={{uuid}}]
|
||||
└─ NO → [等待人工確認]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 錯誤碼
|
||||
|
||||
| HTTP | 說明 |
|
||||
|:---:|------|
|
||||
| 200 | 成功 |
|
||||
| 400 | 缺少 video_uuid 或參數錯誤 |
|
||||
| 401 | API Key 無效 |
|
||||
| 404 | 資源不存在 |
|
||||
| 422 | 請求體缺少 video_uuid |
|
||||
| 500 | 伺服器錯誤 |
|
||||
|
||||
---
|
||||
|
||||
## 資料庫結構
|
||||
|
||||
### person_identities
|
||||
|
||||
| 欄位 | 類型 | 說明 |
|
||||
|------|------|------|
|
||||
| `person_id` | VARCHAR | 識別碼 (每部影片獨立) |
|
||||
| `video_uuid` | VARCHAR | **所屬影片 (必填)** |
|
||||
| `name` | VARCHAR | 人物名稱 |
|
||||
| `speaker_id` | VARCHAR | 對應說話者 ID (每部影片獨立) |
|
||||
| `appearance_count` | INT | 出場次數 |
|
||||
| `is_confirmed` | BOOLEAN | 是否已確認 |
|
||||
|
||||
### 唯一性約束
|
||||
|
||||
```sql
|
||||
UNIQUE (video_uuid, person_id)
|
||||
```
|
||||
|
||||
每部影片可以有自己的 `Person_0`,但同一部影片內 `person_id` 必須唯一。
|
||||
@@ -0,0 +1,183 @@
|
||||
# Face, Speaker, Person, Identity API 教學示範
|
||||
|
||||
本文件將以 1963 年電影《Charade》(謎中謎)為例,示範如何使用 API 管理 **Face** (臉孔)、**Person** (影片中的角色實體) 與 **Identity** (真實身份)。
|
||||
|
||||
## 核心概念定義
|
||||
|
||||
在開始之前,請區分以下名詞:
|
||||
|
||||
1. **Face (臉孔)**: 影像中偵測到的具體臉部特徵數據(向量)。
|
||||
2. **Person (角色實體)**: 在特定影片中出現的角色。他是 Face + Speaker (說話者) 的集合體。
|
||||
* *例如:影片 `384b0ff44aaaa1f1` 中的 `Person_17`。*
|
||||
3. **Identity (真實身份)**: 跨越所有影片的全域實體(如真實演員或新聞人物)。
|
||||
* *例如:Cary Grant, Audrey Hepburn。*
|
||||
|
||||
---
|
||||
|
||||
## 前置準備
|
||||
|
||||
* **API URL**: `http://localhost:3003`
|
||||
* **API Key**: `/`
|
||||
* **目標影片 (Video UUID)**: `384b0ff44aaaa1f1` (Charade)
|
||||
|
||||
---
|
||||
|
||||
## 情境設定
|
||||
|
||||
我們要在影片中識別兩位主角:
|
||||
1. **Audrey Hepburn** (飾演 Reggie Lampert)
|
||||
2. **Cary Grant** (飾演 Peter Joshua)
|
||||
|
||||
---
|
||||
|
||||
## 步驟一:查看影片中的現有角色 (Person List)
|
||||
|
||||
首先,我們查詢系統在影片中偵測到了哪些人物 (Person)。
|
||||
|
||||
```bash
|
||||
curl -s "http://localhost:3003/api/v1/person/list?video_uuid=384b0ff44aaaa1f1&limit=5" \
|
||||
-H "X-API-Key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
**預期回應**:
|
||||
你會看到類似如下的列表,其中包含系統自動分配的 `person_id` (例如 `Person_17`, `Person_4` 等)。
|
||||
|
||||
```json
|
||||
{
|
||||
"persons": [
|
||||
{
|
||||
"person_id": "Person_17",
|
||||
"name": null,
|
||||
"speaker_id": "SPEAKER_1",
|
||||
"appearance_count": 1636
|
||||
},
|
||||
{
|
||||
"person_id": "Person_4",
|
||||
"name": null,
|
||||
"speaker_id": "SPEAKER_0",
|
||||
"appearance_count": 936
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 步驟二:建立身份並綁定角色 (Register Identity from Person)
|
||||
|
||||
假設經過人工確認,我們知道 `Person_17` 是 Audrey Hepburn。我們可以使用單一 API 同時完成 **「建立 Identity」** 與 **「綁定 Person」** 兩個動作。
|
||||
|
||||
### 範例 1: 註冊 Audrey Hepburn
|
||||
|
||||
我們指定 `Person_17` 為 "Audrey Hepburn"。系統會檢查此 Identity 是否存在;若不存在則建立,若已存在則直接綁定。
|
||||
|
||||
```bash
|
||||
curl -s -X POST "http://localhost:3003/api/v1/identities/from-person" \
|
||||
-H "X-API-Key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"video_uuid": "384b0ff44aaaa1f1",
|
||||
"person_id": "Person_17",
|
||||
"identity_name": "Audrey Hepburn",
|
||||
"metadata": { "role": "Reggie Lampert" }
|
||||
}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
**預期回應**:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Successfully registered identity 'Audrey Hepburn' and linked to person 'Person_17'",
|
||||
"identity_id": 10,
|
||||
"identity_name": "Audrey Hepburn",
|
||||
"person_id": "Person_17"
|
||||
}
|
||||
```
|
||||
|
||||
*(註:此操作會自動將該影片中 `Person_17` 的名稱更新為 "Audrey Hepburn")*
|
||||
|
||||
### 範例 2: 註冊 Cary Grant
|
||||
|
||||
假設 `Person_4` 是 Cary Grant,我們進行同樣的操作。
|
||||
|
||||
```bash
|
||||
curl -s -X POST "http://localhost:3003/api/v1/identities/from-person" \
|
||||
-H "X-API-Key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"video_uuid": "384b0ff44aaaa1f1",
|
||||
"person_id": "Person_4",
|
||||
"identity_name": "Cary Grant",
|
||||
"metadata": { "role": "Peter Joshua" }
|
||||
}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
**預期回應**:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Successfully registered identity 'Cary Grant' and linked to person 'Person_4'",
|
||||
"identity_id": 11,
|
||||
"identity_name": "Cary Grant",
|
||||
"person_id": "Person_4"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 步驟三:查看全域身份庫 (List Identities)
|
||||
|
||||
現在我們可以查看所有已建立的「真實身份」,這些身份是跨影片通用的。
|
||||
|
||||
```bash
|
||||
curl -s "http://localhost:3003/api/v1/identities?limit=10" \
|
||||
-H "X-API-Key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
**預期回應**:
|
||||
你應該能看到剛剛建立的 "Audrey Hepburn" 和 "Cary Grant"。
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 11,
|
||||
"name": "Cary Grant",
|
||||
"metadata": { "role": "Peter Joshua" }
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"name": "Audrey Hepburn",
|
||||
"metadata": { "role": "Reggie Lampert" }
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 步驟四:驗證綁定結果
|
||||
|
||||
再次查詢影片中的 `Person` 列表,確認名稱是否已自動更新。
|
||||
|
||||
```bash
|
||||
curl -s "http://localhost:3003/api/v1/person/list?video_uuid=384b0ff44aaaa1f1&limit=5" \
|
||||
-H "X-API-Key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
**預期結果**:
|
||||
原本的 `Person_17` 現在應該顯示為 `"name": "Audrey Hepburn"`。
|
||||
|
||||
---
|
||||
|
||||
## 常見問題 (FAQ)
|
||||
|
||||
**Q: 如果我想把「現有的 Person」綁定到「已經存在的 Identity」,要怎麼做?**
|
||||
A: 使用相同的 `POST /api/v1/identities/from-person` API。只要傳入相同的 `identity_name` (例如 "Audrey Hepburn"),系統會自動找到該 Identity 並將新的 Person 連結過去,不會建立重複的 Identity。
|
||||
|
||||
**Q: Identity 和 Person 的差別是什麼?**
|
||||
A: **Identity** 是真實世界的人(例如 "Tom Hanks"),這是全域共享的。
|
||||
**Person** 是他在某部電影裡的具體出現(例如《阿甘正傳》裡的阿甘)。一個 Identity 可以對應多個影片中的多個 Person。
|
||||
97
docs_v1.0/AI_AGENTS/IDENTITY/FACE_SPEAKER_PERSON_PROGRESS.md
Normal file
97
docs_v1.0/AI_AGENTS/IDENTITY/FACE_SPEAKER_PERSON_PROGRESS.md
Normal file
@@ -0,0 +1,97 @@
|
||||
# Face/Speaker/Person 分析完成度
|
||||
|
||||
**UUID**: `384b0ff44aaaa1f1`
|
||||
**视频**: Charade (1963) - ~115 min, 412,343 frames, 59.94 fps
|
||||
**更新日期**: 2026-04-14
|
||||
|
||||
---
|
||||
|
||||
## 📊 数据统计
|
||||
|
||||
| 模块 | 状态 | 文件 | 数据量 |
|
||||
|------|------|------|--------|
|
||||
| **Face Detection** | ✅ 完成 | `384b0ff44aaaa1f1.face.json` | 10,691 frames, 25,174 faces |
|
||||
| **Face Clustering** | ✅ 完成 | `384b0ff44aaaa1f1.face_clustered.json` | 302 unique Person IDs |
|
||||
| **ASR (语音识别)** | ✅ 完成 | `384b0ff44aaaa1f1.asr.json` | 1,011 segments |
|
||||
| **ASRX (增强语音)** | ✅ 完成 | `384b0ff44aaaa1f1.asrx.json` | - |
|
||||
| **Pose (姿态)** | ✅ 完成 | `384b0ff44aaaa1f1.pose.json` | - |
|
||||
| **Speaker Diarization** | ⚠️ 未集成 | - | ASR segments 无 speaker 信息 |
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Top 20 人物 (按帧数)
|
||||
|
||||
| Person ID | 帧数 | 说明 |
|
||||
|-----------|------|------|
|
||||
| Person_0 | 17,832 | 主角 (Cary Grant/Audrey Hepburn) |
|
||||
| Person_17 | 1,636 | 主要配角 |
|
||||
| Person_4 | 936 | 主要配角 |
|
||||
| Person_25 | 217 | 次要角色 |
|
||||
| Person_12 | 154 | 次要角色 |
|
||||
| Person_46 | 122 | - |
|
||||
| Person_70 | 119 | - |
|
||||
| Person_8 | 109 | - |
|
||||
| Person_3 | 109 | - |
|
||||
| Person_124 | 97 | - |
|
||||
| Person_37 | 95 | - |
|
||||
| Person_176 | 90 | - |
|
||||
| Person_34 | 85 | - |
|
||||
| Person_80 | 78 | - |
|
||||
| Person_50 | 73 | - |
|
||||
| Person_94 | 73 | - |
|
||||
| Person_33 | 63 | - |
|
||||
| Person_21 | 58 | - |
|
||||
| Person_14 | 57 | - |
|
||||
| Person_7 | 57 | - |
|
||||
|
||||
**总计**: 302 个独立 Person ID,其中 282 个出现少于 57 帧。
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 未完成的整合
|
||||
|
||||
### 1. Speaker Diarization (说话者识别)
|
||||
- **问题**: ASR 的 `segments` 中没有 `speaker` 字段
|
||||
- **影响**: 无法将语音片段关联到具体说话者
|
||||
- **待办**:
|
||||
- 运行 speaker diarization 模型
|
||||
- 或使用 ASRX 输出中的 speaker_id
|
||||
|
||||
### 2. Face ↔ Speaker 关联
|
||||
- **脚本存在**: `scripts/sync_face_speaker_to_chunks.py`
|
||||
- **状态**: 需要数据库支持 (chunks 表)
|
||||
- **功能**: 将 face_ids 和 speaker_ids 写入 chunks 表
|
||||
|
||||
### 3. Face ↔ ASR 验证
|
||||
- **文档存在**: `scripts/ASR_FACE_POSE_INTEGRATION.md`
|
||||
- **状态**: 方案设计完成,但未执行
|
||||
- **功能**: 使用 Face + Pose 验证 ASR 语句的置信度
|
||||
|
||||
### 4. 人物命名/识别
|
||||
- **当前**: 只有机器生成的 Person_0, Person_1...
|
||||
- **待办**:
|
||||
- 将主要人物与演员名字关联 (Cary Grant, Audrey Hepburn 等)
|
||||
- 使用 face_registration 功能注册已知演员
|
||||
|
||||
---
|
||||
|
||||
## 📁 相关脚本
|
||||
|
||||
| 脚本 | 用途 | 状态 |
|
||||
|------|------|------|
|
||||
| `face_clustering_processor.py` | 人脸聚类 | ✅ 已执行 |
|
||||
| `fast_face_clustering_processor.py` | 快速人脸聚类 | 备选 |
|
||||
| `sync_face_speaker_to_chunks.py` | 同步到数据库 | 待执行 |
|
||||
| `match_speakers_to_chunks.py` | 匹配说话者 | 待执行 |
|
||||
| `export_person_thumbnails.py` | 导出人物缩略图 | 可用 |
|
||||
| `face_registration.py` | 人脸注册 | 可用 |
|
||||
| `register_sample_faces.py` | 注册样本 | 可用 |
|
||||
|
||||
---
|
||||
|
||||
## 🔧 建议下一步
|
||||
|
||||
1. **检查 ASRX 输出** 是否有 speaker diarization 信息
|
||||
2. **导出 Top 20 人物缩略图** 供人工识别
|
||||
3. **关联主要演员名字** 到 Person_0, Person_17, Person_4 等
|
||||
4. **执行 Face ↔ ASR 验证** 提升语音识别置信度
|
||||
421
docs_v1.0/AI_AGENTS/IDENTITY/FACE_SPEAKER_PERSON_QUICK_START.md
Normal file
421
docs_v1.0/AI_AGENTS/IDENTITY/FACE_SPEAKER_PERSON_QUICK_START.md
Normal file
@@ -0,0 +1,421 @@
|
||||
# Face / Speaker / Person API 簡易指南
|
||||
|
||||
> **版本**: 1.1 | **適用**: 前端開發團隊
|
||||
> **更新日期**: 2026-04-17
|
||||
>
|
||||
> **⚠️ 重要**: 3002 (正式版) 和 3003 (開發版) 使用**完全獨立的資料空間** (public vs dev schema),絕非共用。開發版測試不會影響正式版資料。
|
||||
|
||||
---
|
||||
|
||||
## 快速開始
|
||||
|
||||
```bash
|
||||
export BASE="http://localhost:3002"
|
||||
export KEY="muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69"
|
||||
export UUID="384b0ff44aaaa1f1"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. 用 uuid + chunk_id 查看 face / speaker / person
|
||||
|
||||
### 取得 chunk 內的人物
|
||||
|
||||
```bash
|
||||
curl "$BASE/api/v1/chunks/sentence_0093/persons" \
|
||||
-H "X-API-Key: $KEY"
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"chunk_id": "sentence_0093",
|
||||
"persons": [
|
||||
{
|
||||
"person_id": "Person_0",
|
||||
"name": "Person_0",
|
||||
"confidence": 0.85,
|
||||
"overlap_duration": 3.2
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 取得 chunk 的 speaker(從 content 欄位)
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE/api/v1/search/universal" \
|
||||
-H "X-API-Key: $KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "", "uuid": "'$UUID'", "types": ["chunk"], "filters": {"speaker_id": "SPEAKER_0"}, "limit": 10}'
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"type": "chunk",
|
||||
"chunk_id": "sentence_0093",
|
||||
"chunk_type": "sentence",
|
||||
"start_frame": 29795,
|
||||
"end_frame": 29963,
|
||||
"fps": 59.94,
|
||||
"start_time": 497.08,
|
||||
"end_time": 499.88,
|
||||
"text": "You could have the stamps.",
|
||||
"speaker_id": "SPEAKER_0"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 統一搜尋 chunk + face + person
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE/api/v1/search/universal" \
|
||||
-H "X-API-Key: $KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "stamp", "uuid": "'$UUID'", "types": ["chunk", "person"], "limit": 10}'
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"query": "stamp",
|
||||
"results": [
|
||||
{
|
||||
"type": "chunk",
|
||||
"chunk_id": "sentence_1566",
|
||||
"chunk_type": "sentence",
|
||||
"start_frame": 329980,
|
||||
"end_frame": 330040,
|
||||
"fps": 59.94,
|
||||
"start_time": 5506.84,
|
||||
"end_time": 5507.84,
|
||||
"text": "The envelope, but the stamps on it",
|
||||
"speaker_id": "SPEAKER_0"
|
||||
},
|
||||
{
|
||||
"type": "person",
|
||||
"person_id": "Person_0",
|
||||
"name": "Person_0",
|
||||
"speaker_id": "SPEAKER_0",
|
||||
"appearance_count": 17832
|
||||
}
|
||||
],
|
||||
"total": 10,
|
||||
"took_ms": 27
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. 選擇 face 並綁定 person
|
||||
|
||||
### 步驟 1: 列出所有人物
|
||||
|
||||
```bash
|
||||
curl "$BASE/api/v1/person/list?min_appearances=100&has_speaker=true&limit=20" \
|
||||
-H "X-API-Key: $KEY"
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"persons": [
|
||||
{
|
||||
"person_id": "Person_0",
|
||||
"name": "Person_0",
|
||||
"speaker_id": "SPEAKER_0",
|
||||
"appearance_count": 17832
|
||||
},
|
||||
{
|
||||
"person_id": "Person_17",
|
||||
"name": "Person_17",
|
||||
"speaker_id": "SPEAKER_1",
|
||||
"appearance_count": 1636
|
||||
}
|
||||
],
|
||||
"total": 9
|
||||
}
|
||||
```
|
||||
|
||||
### 步驟 2: 查看人物詳情 + 取得截圖
|
||||
|
||||
```bash
|
||||
# 查看詳情
|
||||
curl "$BASE/api/v1/person/Person_0" -H "X-API-Key: $KEY"
|
||||
|
||||
# 取得臉部截圖
|
||||
curl "$BASE/api/v1/person/Person_0/thumbnail?video_uuid=$UUID" \
|
||||
-H "X-API-Key: $KEY" -o person0_face.jpg
|
||||
|
||||
# 取得第 5 次出現的臉部截圖
|
||||
curl "$BASE/api/v1/person/Person_0/thumbnail?video_uuid=$UUID&index=4" \
|
||||
-H "X-API-Key: $KEY" -o person0_face_5.jpg
|
||||
```
|
||||
|
||||
### 步驟 3: 綁定名稱(將 face 關聯到 person)
|
||||
|
||||
```bash
|
||||
curl -X PATCH "$BASE/api/v1/person/Person_0" \
|
||||
-H "X-API-Key: $KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name": "Cary Grant", "is_confirmed": true}'
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Person 'Cary Grant' updated successfully",
|
||||
"person_id": "Person_0"
|
||||
}
|
||||
```
|
||||
|
||||
### 步驟 4: 註冊新臉孔(建立參考樣本)
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE/api/v1/face/register" \
|
||||
-H "X-API-Key: $KEY" \
|
||||
-F "image=@known_face.jpg" \
|
||||
-F "name=Cary Grant" \
|
||||
-F 'metadata={"imdb_id": "nm0000001"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 合併前檢視:取得臉部截圖
|
||||
|
||||
### 取得單張截圖
|
||||
|
||||
```bash
|
||||
# 預設:第一次出現的臉部
|
||||
curl "$BASE/api/v1/person/Person_0/thumbnail?video_uuid=$UUID" \
|
||||
-H "X-API-Key: $KEY" -o face.jpg
|
||||
|
||||
# 指定第 N 次出現
|
||||
curl "$BASE/api/v1/person/Person_0/thumbnail?video_uuid=$UUID&index=10" \
|
||||
-H "X-API-Key: $KEY" -o face_10.jpg
|
||||
```
|
||||
|
||||
### 找出相似人物(可能為同一人)
|
||||
|
||||
```bash
|
||||
curl "$BASE/api/v1/person/Person_0/similar?threshold=0.5&limit=10" \
|
||||
-H "X-API-Key: $KEY"
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"person_id": "Person_0",
|
||||
"similar_persons": [
|
||||
{
|
||||
"person_id": "Person_4",
|
||||
"name": "Person_4",
|
||||
"speaker_id": "SPEAKER_0",
|
||||
"similarity": 0.7
|
||||
},
|
||||
{
|
||||
"person_id": "Person_25",
|
||||
"name": "Person_25",
|
||||
"speaker_id": "SPEAKER_0",
|
||||
"similarity": 0.7
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 取得 AI 合併建議
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE/api/v1/person/suggest" \
|
||||
-H "X-API-Key: $KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"video_uuid": "'$UUID'"}'
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"merge_suggestions": [
|
||||
{
|
||||
"person_id": "Person_0",
|
||||
"merge_with": ["Person_4", "Person_25"],
|
||||
"confidence": 0.65,
|
||||
"reasons": [
|
||||
"All share speaker_id: SPEAKER_0",
|
||||
"Primary Person_0 has 17832 appearances (89% of group)"
|
||||
],
|
||||
"action": "needs_review"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 統一搜尋
|
||||
|
||||
### ⚠️ 重要:搜尋 chunks 時 uuid 為必填
|
||||
|
||||
**只有 `uuid + chunk_id` 組合才是唯一識別碼。** 單獨 `chunk_id` 在不同影片中可能重複。
|
||||
|
||||
```bash
|
||||
# ✅ 正確:包含 uuid
|
||||
curl -X POST "$BASE/api/v1/search/universal" \
|
||||
-H "X-API-Key: $KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "stamp", "uuid": "'$UUID'", "types": ["chunk"]}'
|
||||
|
||||
# ❌ 錯誤:缺少 uuid
|
||||
curl -X POST "$BASE/api/v1/search/universal" \
|
||||
-H "X-API-Key: $KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "stamp", "types": ["chunk"]}'
|
||||
# 回傳: {"error": "uuid is required for chunk search"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 使用 API 合併 face / speaker / person
|
||||
|
||||
### ⚠️ 重要:合併撤銷限制
|
||||
|
||||
**合併撤銷完全依賴 `merge_history` 記錄。**
|
||||
|
||||
| 情況 | 可否撤銷 |
|
||||
|------|:---:|
|
||||
| 使用 `POST /api/v1/person/merge` API 合併 | ✅ 可以(自動記錄歷史) |
|
||||
| 手動修改資料庫合併 | ❌ 不可以(無歷史記錄) |
|
||||
| 舊版程式碼合併(無 merge_history 表) | ❌ 不可以 |
|
||||
| 已撤銷過的合併 | ❌ 不可以(防止重複撤銷) |
|
||||
|
||||
**每次合併 API 都會回傳 `merge_id`,請務必儲存以便日後撤銷。**
|
||||
|
||||
### 執行合併
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE/api/v1/person/merge" \
|
||||
-H "X-API-Key: $KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"target_person_id": "Person_0",
|
||||
"source_person_ids": ["Person_4", "Person_25"]
|
||||
}'
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Merged 2 persons into Person_0",
|
||||
"target_person_id": "Person_0",
|
||||
"merge_id": "5b12e3ac-12fa-45c0-88e1-5cff67604a7d"
|
||||
}
|
||||
```
|
||||
|
||||
### 合併做了什麼?
|
||||
|
||||
```
|
||||
合併前:
|
||||
Person_0 (17832 幀, SPEAKER_0)
|
||||
Person_4 (936 幀, SPEAKER_0)
|
||||
Person_25 (217 幀, SPEAKER_0)
|
||||
|
||||
合併後:
|
||||
Person_0 (17832+936+217=18985 幀, SPEAKER_0) ← 保留
|
||||
Person_4 ← 刪除
|
||||
Person_25 ← 刪除
|
||||
```
|
||||
|
||||
### 撤銷合併
|
||||
|
||||
```bash
|
||||
# 使用合併時回傳的 merge_id
|
||||
curl -X POST "$BASE/api/v1/person/merge/undo" \
|
||||
-H "X-API-Key: $KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"merge_id": "5b12e3ac-12fa-45c0-88e1-5cff67604a7d"}'
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Undo merge completed. Restored 2 source persons",
|
||||
"merge_id": "5b12e3ac-12fa-45c0-88e1-5cff67604a7d",
|
||||
"target_person_id": "Person_0",
|
||||
"restored_persons": ["Person_4", "Person_25"]
|
||||
}
|
||||
```
|
||||
|
||||
**⚠️ 如果沒有 merge_id(手動合併/舊版合併),無法撤銷。**
|
||||
|
||||
### 查看合併歷史
|
||||
|
||||
```bash
|
||||
curl "$BASE/api/v1/person/merge/history" -H "X-API-Key: $KEY"
|
||||
```
|
||||
|
||||
### 完整合併流程
|
||||
|
||||
```
|
||||
1. 取得建議 → POST /api/v1/person/suggest
|
||||
2. 檢視截圖 → GET /api/v1/person/:id/thumbnail
|
||||
3. 檢視相似 → GET /api/v1/person/:id/similar
|
||||
4. 執行合併 → POST /api/v1/person/merge ← 儲存 merge_id!
|
||||
5. 確認結果 → GET /api/v1/person/list
|
||||
6. 如需撤銷 → POST /api/v1/person/merge/undo ← 需要 merge_id
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API 速查表
|
||||
|
||||
| 用途 | 方法 | 端點 |
|
||||
|------|:---:|------|
|
||||
| **查看 chunk 內人物** | GET | `/api/v1/chunks/:chunk_id/persons` |
|
||||
| **搜尋人物** | GET | `/api/v1/search/persons?query=Person` |
|
||||
| **列出人物** | GET | `/api/v1/person/list?limit=20` |
|
||||
| **人物詳情** | GET | `/api/v1/person/:id` |
|
||||
| **人物截圖** | GET | `/api/v1/person/:id/thumbnail?video_uuid=...` |
|
||||
| **相似人物** | GET | `/api/v1/person/:id/similar` |
|
||||
| **AI 建議** | POST | `/api/v1/person/suggest` |
|
||||
| **綁定名稱** | PATCH | `/api/v1/person/:id` |
|
||||
| **合併人物** | POST | `/api/v1/person/merge` |
|
||||
| **撤銷合併** | POST | `/api/v1/person/merge/undo` |
|
||||
| **合併歷史** | GET | `/api/v1/person/merge/history` |
|
||||
| **統一搜尋** | POST | `/api/v1/search/universal` |
|
||||
| **註冊臉孔** | POST | `/api/v1/face/register` |
|
||||
|
||||
---
|
||||
|
||||
## 錯誤處理
|
||||
|
||||
```bash
|
||||
# 錯誤回應
|
||||
curl -X POST "$BASE/api/v1/person/merge" \
|
||||
-H "X-API-Key: $KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"target_person_id": "Person_0", "source_person_ids": []}'
|
||||
# → "source_person_ids cannot be empty"
|
||||
```
|
||||
|
||||
| 狀態碼 | 說明 |
|
||||
|:---:|------|
|
||||
| 200 | 成功 |
|
||||
| 400 | 參數錯誤 |
|
||||
| 401 | API Key 無效 |
|
||||
| 404 | 找不到 |
|
||||
| 500 | 伺服器錯誤 |
|
||||
|
||||
---
|
||||
|
||||
## 資料修正
|
||||
|
||||
發現綁定錯誤時,參考 [人物資料修正機制指南](./PERSON_CORRECTION_GUIDE.md)
|
||||
|
||||
| 錯誤類型 | 修正方式 |
|
||||
|---------|---------|
|
||||
| Speaker 綁錯 | `POST /person/:id/reassign-speaker` |
|
||||
| 不該綁 Speaker | `POST /person/:id/unbind-speaker` |
|
||||
| Appearance 分錯人 | `POST /person/:id/reassign-appearance` |
|
||||
| 錯誤 Appearance | `POST /person/:id/remove-appearance` |
|
||||
| 兩人被合併為一 | `POST /person/:id/split` |
|
||||
| 錯誤合併 | `POST /person/merge/undo` |
|
||||
| 錯誤命名 | `PATCH /person/:id` |
|
||||
167
docs_v1.0/AI_AGENTS/IDENTITY/FACE_SPEAKER_PERSON_WORKFLOW.md
Normal file
167
docs_v1.0/AI_AGENTS/IDENTITY/FACE_SPEAKER_PERSON_WORKFLOW.md
Normal file
@@ -0,0 +1,167 @@
|
||||
# Face / Speaker / Person / Identity Workflow Guide
|
||||
|
||||
This document describes the end-to-end workflow for managing characters in Momentry Core, from raw detection to a clean, aggregated identity database.
|
||||
|
||||
## 📊 1. Workflow Visualization
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
%% Nodes
|
||||
Start((Start Analysis))
|
||||
ListPersons[List Persons]
|
||||
|
||||
subgraph "Phase 1: Registration"
|
||||
CheckIdentity{Identity Exists?}
|
||||
Register[Register Identity]
|
||||
Link[Link Person to Identity]
|
||||
end
|
||||
|
||||
subgraph "Phase 2: Aggregation"
|
||||
Suggest[Get AI Suggestions]
|
||||
Review[Review Suggestions]
|
||||
Merge[Execute Merge]
|
||||
Confirm[Confirm Result]
|
||||
end
|
||||
|
||||
End((Database Clean))
|
||||
|
||||
%% Flow
|
||||
Start --> ListPersons
|
||||
ListPersons --> CheckIdentity
|
||||
|
||||
CheckIdentity -- No --> Register
|
||||
Register --> Link
|
||||
Link --> Suggest
|
||||
|
||||
CheckIdentity -- Yes --> Suggest
|
||||
|
||||
Suggest --> Review
|
||||
Review -- Merge Recommended --> Merge
|
||||
Review -- Naming Recommended --> Rename[Update Name]
|
||||
Rename --> Confirm
|
||||
|
||||
Merge --> Confirm
|
||||
Confirm --> End
|
||||
|
||||
style Start fill:#f9f,stroke:#333
|
||||
style End fill:#bbf,stroke:#333
|
||||
style Register fill:#dfd,stroke:#333
|
||||
style Merge fill:#dfd,stroke:#333
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ 2. Step-by-Step API Operations
|
||||
|
||||
### Phase 1: Registration (Creating Identities)
|
||||
**Scenario**: You see `Person_17` is Audrey Hepburn. You want to create a global record for her.
|
||||
|
||||
1. **Find the Person**:
|
||||
```bash
|
||||
curl -s "http://localhost:3003/api/v1/person/list?video_uuid=...&limit=5" ...
|
||||
# Output: Person_17 (1636 frames, null name)
|
||||
```
|
||||
|
||||
2. **Register Identity**:
|
||||
```bash
|
||||
curl -X POST "http://localhost:3003/api/v1/identities/from-person" ... \
|
||||
-d '{
|
||||
"video_uuid": "...",
|
||||
"person_id": "Person_17",
|
||||
"identity_name": "Audrey Hepburn"
|
||||
}'
|
||||
```
|
||||
*Result: `Person_17` is now named "Audrey Hepburn". A global `identity_id` is created.*
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Suggestion (AI Analysis)
|
||||
**Scenario**: You suspect `Person_25` might also be Audrey Hepburn, or you just want to clean up the data.
|
||||
|
||||
1. **Ask for Suggestions**:
|
||||
```bash
|
||||
curl -X POST "http://localhost:3003/api/v1/person/suggest" ... \
|
||||
-d '{"video_uuid": "..."}'
|
||||
```
|
||||
*Response*:
|
||||
```json
|
||||
{
|
||||
"merge_suggestions": [
|
||||
{
|
||||
"person_id": "Person_17",
|
||||
"merge_with": ["Person_25"],
|
||||
"reasons": ["All share speaker_id: SPEAKER_1", "Person_17 has 88% of frames"],
|
||||
"action": "auto_apply"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Review & Execution
|
||||
**Scenario**: You verify the suggestion. The AI logic (Shared Speaker + Frame dominance) seems correct.
|
||||
|
||||
1. **Execute the Merge**:
|
||||
```bash
|
||||
curl -X POST "http://localhost:3003/api/v1/person/merge" ... \
|
||||
-d '{
|
||||
"video_uuid": "...",
|
||||
"target_person_id": "Person_17",
|
||||
"source_person_ids": ["Person_25"]
|
||||
}'
|
||||
```
|
||||
*Result*: `Person_25` is deleted. All 217 frames of `Person_25` are added to `Person_17`.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 3. Automated Demo Script
|
||||
|
||||
Run the following script to see the entire process in action automatically.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# scripts/demo_identity_workflow.sh
|
||||
# Usage: chmod +x scripts/demo_identity_workflow.sh && ./scripts/demo_identity_workflow.sh
|
||||
|
||||
API_URL="http://localhost:3002"
|
||||
API_KEY="muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69"
|
||||
UUID="384b0ff44aaaa1f1"
|
||||
|
||||
echo "🎬 === MOMENTRY IDENTITY WORKFLOW DEMO ==="
|
||||
|
||||
# 1. Registration
|
||||
echo "👉 STEP 1: Registering Person_17 as Audrey Hepburn..."
|
||||
curl -s -X POST "$API_URL/api/v1/identities/from-person" \
|
||||
-H "X-API-Key: $API_KEY" -H "Content-Type: application/json" \
|
||||
-d "{\"video_uuid\":\"$UUID\", \"person_id\":\"Person_17\", \"identity_name\":\"Audrey Hepburn\"}" \
|
||||
| python3 -m json.tool
|
||||
|
||||
# 2. Suggestion
|
||||
echo ""
|
||||
echo "👉 STEP 2: Asking AI for cleaning suggestions..."
|
||||
curl -s -X POST "$API_URL/api/v1/person/suggest" \
|
||||
-H "X-API-Key: $API_KEY" -H "Content-Type: application/json" \
|
||||
-d "{\"video_uuid\":\"$UUID\"}" \
|
||||
| python3 -c "
|
||||
import sys, json
|
||||
d = json.load(sys.stdin)
|
||||
sugs = d.get('naming_suggestions', []) + d.get('merge_suggestions', [])
|
||||
if sugs:
|
||||
print(f' Found {len(sugs)} suggestions.')
|
||||
for s in sugs:
|
||||
print(f' - {s}')
|
||||
else:
|
||||
print(' No suggestions (Data is already clean!).')
|
||||
"
|
||||
|
||||
# 3. Execution (Example Merge if Person_25 existed)
|
||||
echo ""
|
||||
echo "👉 STEP 3: Simulating a merge (Merging hypothetical Person_25 -> Person_17)..."
|
||||
# Note: In a real scenario, Person_25 would exist.
|
||||
# Here we just show the command structure.
|
||||
echo " Command: POST /api/v1/person/merge { target: 'Person_17', sources: ['Person_25'] }"
|
||||
echo " Result: Person_25 frames added to Person_17. Person_25 deleted."
|
||||
|
||||
echo ""
|
||||
echo "✅ Demo Complete."
|
||||
214
docs_v1.0/AI_AGENTS/IDENTITY/IDENTITY_MANAGEMENT_API.md
Normal file
214
docs_v1.0/AI_AGENTS/IDENTITY/IDENTITY_MANAGEMENT_API.md
Normal file
@@ -0,0 +1,214 @@
|
||||
# 📘 Momentry 身份管理 (Identity Management) API 實作指南
|
||||
|
||||
本文件示範如何透過 API 完成「從影片選擇 → 臉部分析 → 全域身份註冊」的完整流程。
|
||||
|
||||
## 1. 選擇目標影片
|
||||
|
||||
**目標**: 獲取系統中已註冊的影片列表,選擇要進行管理的影片。
|
||||
|
||||
**API**: `GET /api/v1/videos`
|
||||
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:3002/api/v1/videos" \
|
||||
-H "x-api-key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" | jq .
|
||||
```
|
||||
|
||||
**回應範例**:
|
||||
```json
|
||||
{
|
||||
"videos": [
|
||||
{
|
||||
"uuid": "384b0ff44aaaa1f1",
|
||||
"file_name": "Old_Time_Movie_Show_-_Charade_1963.HD.mov",
|
||||
"duration": 6879.33
|
||||
},
|
||||
{
|
||||
"uuid": "9760d0820f0cf9a7",
|
||||
"file_name": "ExaSAN PCIe series - Director Ou.mp4",
|
||||
"duration": 159.64
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
> **決策**: 我們選擇 `Charade 1963` (UUID: `384b0ff44aaaa1f1`) 進行管理。
|
||||
|
||||
---
|
||||
|
||||
## 2. 分析影片內的所有人物 (Faces / Persons / Speakers)
|
||||
|
||||
**目標**: 查看該影片內所有偵測到的「臉群 (Clusters)」。區分**已命名 (Named)**、**待命名 (Unregistered)** 與 **AI 建議**。
|
||||
|
||||
**API**: `GET /api/v1/videos/{uuid}/faces`
|
||||
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:3002/api/v1/videos/384b0ff44aaaa1f1/faces" \
|
||||
-H "x-api-key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" | jq .
|
||||
```
|
||||
|
||||
**回應範例**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"video_uuid": "384b0ff44aaaa1f1",
|
||||
"total_faces": 6,
|
||||
"registered_count": 0,
|
||||
"unregistered_count": 6,
|
||||
"clusters": [
|
||||
{
|
||||
"cluster_id": "Person_4",
|
||||
"face_count": 45,
|
||||
"status": "unregistered",
|
||||
"identity": {
|
||||
"name": "Cary Grant",
|
||||
"is_confirmed": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"cluster_id": "Person_17",
|
||||
"face_count": 32,
|
||||
"status": "unregistered",
|
||||
"identity": {
|
||||
"name": "Audrey Hepburn",
|
||||
"is_confirmed": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"cluster_id": "Person_12",
|
||||
"face_count": 10,
|
||||
"status": "unregistered",
|
||||
"identity": { "name": "Person_12" }
|
||||
},
|
||||
{
|
||||
"cluster_id": "Person_124",
|
||||
"face_count": 5,
|
||||
"status": "unregistered",
|
||||
"identity": null
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 如何解讀結果?
|
||||
|
||||
| 欄位 | 說明 | 狀態 |
|
||||
| :--- | :--- | :--- |
|
||||
| **`identity.name`** | 若顯示具體人名 (如 "Audrey Hepburn"),代表 **已命名**。 | ✅ 待註冊 |
|
||||
| **`identity.name`** | 若顯示 `Person_XX` (系統預設名),代表 **待命名**。 | 🔄 等待 AI 或人工命名 |
|
||||
| **`identity: null`** | 代表完全 **未識別**,通常數量較少。 | ❓ 待處理 |
|
||||
|
||||
---
|
||||
|
||||
## 3. 註冊全域身份 (Register Identity)
|
||||
|
||||
**目標**: 將已命名的人物升級為 **全域身份 (Global Identity)**。這能讓系統在其他影片中自動認出他們。
|
||||
|
||||
**API**: `POST /api/v1/person/{person_id}/register?video_uuid={uuid}`
|
||||
|
||||
### 3.1 註冊 Audrey Hepburn
|
||||
|
||||
```bash
|
||||
curl -s -X POST "http://127.0.0.1:3002/api/v1/person/Person_17/register?video_uuid=384b0ff44aaaa1f1" \
|
||||
-H "x-api-key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" | jq .
|
||||
```
|
||||
|
||||
**回應**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Successfully registered as global identity",
|
||||
"person_id": "Person_17",
|
||||
"name": "Audrey Hepburn",
|
||||
"face_identity_id": 12
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 註冊 Cary Grant
|
||||
|
||||
```bash
|
||||
curl -s -X POST "http://127.0.0.1:3002/api/v1/person/Person_4/register?video_uuid=384b0ff44aaaa1f1" \
|
||||
-H "x-api-key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" | jq .
|
||||
```
|
||||
|
||||
**回應**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"face_identity_id": 13,
|
||||
"name": "Cary Grant"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ✅ 驗證成果
|
||||
|
||||
現在可以使用全域搜尋 API 確認身份是否註冊成功:
|
||||
|
||||
```bash
|
||||
curl -s -X POST "http://127.0.0.1:3002/api/v1/identities/search" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "x-api-key: muser_..." \
|
||||
-d '{"query": "Audrey"}' | jq '.identities[] | {name: .profile.name, identity_id: .face_identity_id}'
|
||||
```
|
||||
|
||||
**結果**:
|
||||
```json
|
||||
{
|
||||
"name": "Audrey Hepburn",
|
||||
"identity_id": 12
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 擷取身份 / 人物 / 臉部 截圖
|
||||
|
||||
**目標**: 取得特定人物的臉部特寫截圖。
|
||||
由於「Identity (全域身份)」是由多個影片中的「Person (區域人物)」組成,而「Person」是由多個「Face (臉部偵測點)」聚合而成,因此擷取截圖的核心是取得 **該人物在某部影片中的某幀臉部影像**。
|
||||
|
||||
**API**: `GET /api/v1/person/{person_id}/thumbnail`
|
||||
|
||||
### 參數說明
|
||||
|
||||
| 參數 | 類型 | 必填 | 說明 |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `person_id` | Path | ✅ | 人物 ID (例如: `Person_17`) |
|
||||
| `video_uuid` | Query | ✅ | 影片 UUID (用來定位影像源) |
|
||||
| `index` | Query | ❌ | 指定第幾張臉 (預設 `0`) |
|
||||
|
||||
### 4.1 擷取 Audrey Hepburn 的臉部截圖 (預設第一張)
|
||||
|
||||
此指令會自動從 `Charade 1963` 影片中擷取 Audrey Hepburn 最清晰的一張臉,並儲存為 `audrey.jpg`。
|
||||
|
||||
```bash
|
||||
curl -s -o audrey.jpg \
|
||||
"http://127.0.0.1:3002/api/v1/person/Person_17/thumbnail?video_uuid=384b0ff44aaaa1f1" \
|
||||
-H "x-api-key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69"
|
||||
```
|
||||
|
||||
> **注意**: 回應是 **圖片二進位資料 (JPG)**,請使用 `-o filename.jpg` 儲存,**不要**使用 `| jq`。
|
||||
|
||||
### 4.2 擷取 Cary Grant 的其他臉部截圖 (指定 Index)
|
||||
|
||||
若你想看同一人物的其他角度,可以調整 `index` 參數。
|
||||
假設 Cary Grant (`Person_4`) 在影片中出現了 45 次:
|
||||
|
||||
```bash
|
||||
# 擷取第 5 次出現的臉部截圖 (index 從 0 開始)
|
||||
curl -s -o cary_face_5.jpg \
|
||||
"http://127.0.0.1:3002/api/v1/person/Person_4/thumbnail?video_uuid=384b0ff44aaaa1f1&index=4" \
|
||||
-H "x-api-key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69"
|
||||
```
|
||||
|
||||
### 4.3 Identity (全域身份) 的截圖策略
|
||||
|
||||
由於全域 Identity (`face_identity_id: 12`) 跨越多部影片,要取得它的截圖,請先查詢它所屬的影片:
|
||||
|
||||
1. **查詢 Identity 所在的影片**:
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:3002/api/v1/identities/12/videos" \
|
||||
-H "x-api-key: muser_..." | jq '.videos[0].video_uuid'
|
||||
```
|
||||
2. **取得該影片中的對應 Person ID**: 從上一步結果中找到 `person_id` (例如 `Person_17`)。
|
||||
3. **呼叫截圖 API**: 使用該 `video_uuid` 和 `person_id` 呼叫上述截圖 API。
|
||||
|
||||
139
docs_v1.0/AI_AGENTS/SEARCH/SEARCH_PROMPTS.md
Normal file
139
docs_v1.0/AI_AGENTS/SEARCH/SEARCH_PROMPTS.md
Normal file
@@ -0,0 +1,139 @@
|
||||
---
|
||||
document_type: "reference_doc"
|
||||
service: "MOMENTRY_CORE"
|
||||
title: "搜尋範例 Prompt"
|
||||
date: "2026-04-25"
|
||||
version: "V1.0"
|
||||
status: "active"
|
||||
owner: "Warren"
|
||||
created_by: "OpenCode"
|
||||
tags:
|
||||
- "prompt"
|
||||
- "搜尋範例"
|
||||
ai_query_hints:
|
||||
- "查詢 搜尋範例 Prompt 的內容"
|
||||
- "搜尋範例 Prompt 的主要目的是什麼?"
|
||||
- "如何操作或實施 搜尋範例 Prompt?"
|
||||
---
|
||||
|
||||
# 搜尋範例 Prompt
|
||||
|
||||
## 基本搜尋測試
|
||||
|
||||
### 1. 簡單關鍵字搜尋
|
||||
```bash
|
||||
curl -X POST http://localhost:3002/api/v1/search \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "charade", "limit": 5}'
|
||||
```
|
||||
|
||||
### 2. 電影相關詞
|
||||
```
|
||||
charade
|
||||
woody allen
|
||||
audrey hepburn
|
||||
classic movie
|
||||
old time movie
|
||||
romantic comedy
|
||||
```
|
||||
|
||||
### 3. 場景描述
|
||||
```
|
||||
widowed woman
|
||||
secret agent
|
||||
chase scene
|
||||
paris
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 進階搜尋測試
|
||||
|
||||
### 4. 短語搜尋
|
||||
```bash
|
||||
curl -X POST http://localhost:3002/api/v1/search \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "fun plot twists", "limit": 3}'
|
||||
```
|
||||
|
||||
### 5. 情感/描述詞
|
||||
```
|
||||
charming performances
|
||||
hilarious
|
||||
suspenseful
|
||||
dramatic
|
||||
```
|
||||
|
||||
### 6. 動作場景
|
||||
```
|
||||
running
|
||||
chase
|
||||
fighting
|
||||
dancing
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 整合範例
|
||||
|
||||
### n8n Workflow
|
||||
```
|
||||
搜尋詞: "charade"
|
||||
→ 取得 chunk 的 start_time, end_time
|
||||
→ 組裝成影片 URL
|
||||
→ 回傳給用戶
|
||||
```
|
||||
|
||||
### PHP 範例
|
||||
```php
|
||||
$searchTerms = ['charade', 'woody', 'audrey', 'classic'];
|
||||
|
||||
// 搜尋每個詞
|
||||
foreach ($searchTerms as $term) {
|
||||
$ch = curl_init('http://localhost:3002/api/v1/search');
|
||||
curl_setopt($ch, CURLOPT_POST, true);
|
||||
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([
|
||||
'query' => $term,
|
||||
'limit' => 5
|
||||
]));
|
||||
curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']);
|
||||
$response = curl_exec($ch);
|
||||
$data = json_decode($response, true);
|
||||
|
||||
// 處理結果
|
||||
foreach ($data['results'] as $result) {
|
||||
echo "{$result['text']} (score: {$result['score']})\n";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 預期回傳格式
|
||||
|
||||
```json
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"uuid": "a1b10138a6bbb0cd",
|
||||
"chunk_id": "sentence_0006",
|
||||
"chunk_type": "sentence",
|
||||
"start_time": 48.8,
|
||||
"end_time": 55.44,
|
||||
"text": "fun plot twists, Woody Dialog and charming performances...",
|
||||
"score": 0.526
|
||||
}
|
||||
],
|
||||
"query": "charade"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 測試檢查清單
|
||||
|
||||
- [ ] 基本關鍵字搜尋
|
||||
- [ ] n8n 整合格式
|
||||
- [ ] 影片時戳取得
|
||||
- [ ] 多筆結果排序
|
||||
- [ ] 不同 chunk_type 搜尋
|
||||
231
docs_v1.0/AI_AGENTS/SUMMARIZATION/CHUNK_RULE_4_SUMMARY.md
Normal file
231
docs_v1.0/AI_AGENTS/SUMMARIZATION/CHUNK_RULE_4_SUMMARY.md
Normal file
@@ -0,0 +1,231 @@
|
||||
---
|
||||
document_type: "architecture_design"
|
||||
service: "MOMENTRY_CORE"
|
||||
title: "Momentry Core Chunk Rule 4: 摘要分析級檢索 (Summary 5W1H Chunk) (v1.0)"
|
||||
date: "2026-04-21"
|
||||
version: "V1.0"
|
||||
status: "active"
|
||||
owner: "Warren"
|
||||
created_by: "OpenCode"
|
||||
tags:
|
||||
- "momentry"
|
||||
- "core"
|
||||
- "摘要分析級檢索"
|
||||
- "rule"
|
||||
ai_query_hints:
|
||||
- "查詢 Momentry Core Chunk Rule 4: 摘要分析級檢索 (Summary 5W1H Chunk) (v1.0) 的內容"
|
||||
- "Momentry Core Chunk Rule 4: 摘要分析級檢索 (Summary 5W1H Chunk) (v1.0) 的主要目的是什麼?"
|
||||
- "如何操作或實施 Momentry Core Chunk Rule 4: 摘要分析級檢索 (Summary 5W1H Chunk) (v1.0)?"
|
||||
---
|
||||
|
||||
# Momentry Core Chunk Rule 4: 摘要分析級檢索 (Summary 5W1H Chunk) (v1.0)
|
||||
|
||||
| 項目 | 內容 |
|
||||
|------|------|
|
||||
| 建立者 | OpenCode |
|
||||
| 建立時間 | 2026-04-21 |
|
||||
| 文件版本 | V1.0 |
|
||||
|
||||
---
|
||||
|
||||
## 版本歷史
|
||||
|
||||
| 版本 | 日期 | 目的 | 操作人 | 工具/模型 |
|
||||
|------|------|------|--------|-----------|
|
||||
| V1.0 | 2026-04-21 | 定義 Rule 4: 基於 LLM 5W1H 分析的最高層級摘要結構 | OpenCode | OpenCode / Qwen3.6-Plus |
|
||||
|
||||
---
|
||||
|
||||
## 0. 設計目標
|
||||
|
||||
**Rule 4** 的核心概念是**「情節理解」(Storyline Understanding)**。透過將多個場景 (Rule 3) 聚合,並利用大型語言模型 (Gemma4) 進行深度分析,提取 5W1H 結構化資訊,使系統能夠回答複雜的「情節相關問題」。
|
||||
|
||||
- **核心原則**: 5-10 個場景 (Rule 3) = 1 個摘要區塊 (Summary Chunk)。
|
||||
- **結構**: 頂層 Parent Chunk。
|
||||
- **特徵**: 包含 LLM 生成的完整摘要與 **5W1H** (Who, What, When, Where, Why, How) 分析結果。
|
||||
- **優勢**: 支援宏觀劇情檢索、人物動線追蹤與複雜問答 (RAG)。
|
||||
|
||||
---
|
||||
|
||||
## 1. 數據源與聚合邏輯
|
||||
|
||||
Rule 4 是處理管線的終點,依賴 **Rule 3** 的產出以及 **LLM 服務**。
|
||||
|
||||
1. **Rule 3 Chunks (Primary)**: 提供場景級的文本摘要與元數據。
|
||||
- *聚合策略*: 將連續的 5-10 個 Rule 3 Chunks 視為一個「敘事區塊」。
|
||||
2. **LLM Processor (Gemma4)**:
|
||||
- *任務*: 讀取該區塊內所有 Rule 3 的摘要與 ASR 文本。
|
||||
- *輸出*:
|
||||
- **Summary**: 流暢的劇情描述。
|
||||
- **5W1H**: 結構化的關鍵要素提取。
|
||||
3. **Visual/Audio Retention**:
|
||||
- 保留區塊內所有出現過的 `face_ids` (Who) 和 `objects` (What/Where)。
|
||||
|
||||
---
|
||||
|
||||
## 2. Chunk 結構定義
|
||||
|
||||
### 2.1 資料庫結構 (PostgreSQL)
|
||||
|
||||
```sql
|
||||
CREATE TABLE chunks_rule4 (
|
||||
id UUID PRIMARY KEY,
|
||||
asset_uuid UUID NOT NULL,
|
||||
chunk_type VARCHAR(20) DEFAULT 'summary',
|
||||
|
||||
-- 時間軸 (繼承自第一個與最後一個 Rule 3 子區塊)
|
||||
start_frame INT NOT NULL,
|
||||
end_frame INT NOT NULL,
|
||||
start_time_sec DOUBLE PRECISION,
|
||||
end_time_sec DOUBLE PRECISION,
|
||||
|
||||
-- LLM 生成內容
|
||||
summary TEXT NOT NULL, -- 劇情摘要
|
||||
analysis_5w1h JSONB, -- 結構化分析結果
|
||||
|
||||
-- 聚合元數據
|
||||
faces JSONB, -- 區塊內所有人物
|
||||
objects JSONB, -- 區塊內重要物件
|
||||
|
||||
-- 向量索引
|
||||
embedding vector(768), -- 摘要與 5W1H 的混合向量
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- 關聯子區塊
|
||||
ALTER TABLE parent_chunks ADD COLUMN rule4_parent_id UUID REFERENCES chunks_rule4(id);
|
||||
```
|
||||
|
||||
### 2.2 5W1H 結構 (JSONB)
|
||||
|
||||
```json
|
||||
{
|
||||
"who": ["Cary Grant", "Audrey Hepburn"], // 主要人物 (對應 Face ID)
|
||||
"what": ["Searching for the stamps", "Car chase"], // 核心事件
|
||||
"where": ["Paris", "Bank", "Car"], // 地點/場景 (對應 Visual Objects)
|
||||
"when": "Night", // 時間背景 (對應 Time of day)
|
||||
"why": "To pay off a debt", // 動機
|
||||
"how": "By sneaking into the vault" // 手段/過程
|
||||
}
|
||||
```
|
||||
|
||||
### 2.3 JSON 產出範例
|
||||
|
||||
```json
|
||||
{
|
||||
"chunk_id": "550e...0004",
|
||||
"type": "summary",
|
||||
"summary": "Peter 和 Regina 計劃潛入銀行金庫尋找郵票。他們在夜間開車前往,途中遭遇巡邏隊盤查,但最終利用機智脫身。",
|
||||
"start_frame": 5000,
|
||||
"end_frame": 8000,
|
||||
"analysis_5w1h": {
|
||||
"who": ["peter_joshua", "regina_lampert"],
|
||||
"what": ["heist_planning", "evasion"],
|
||||
"where": ["car", "street", "bank_exterior"],
|
||||
"when": "night",
|
||||
"why": "retrieve_stamps",
|
||||
"how": "stealth_deception"
|
||||
},
|
||||
"metadata": {
|
||||
"rule3_count": 7
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 搜尋能力定義
|
||||
|
||||
Rule 4 是 **RAG (Retrieval-Augmented Generation)** 的核心數據源。
|
||||
|
||||
### 3.1 劇情摘要搜尋 (Plot Search)
|
||||
* **場景**: "這部片在講什麼?"、"他們找到郵票了嗎?"
|
||||
* **邏輯**:
|
||||
1. 搜尋 `summary` 向量。
|
||||
2. 返回包含該情節的完整摘要區塊。
|
||||
|
||||
### 3.2 5W1H 結構化查詢 (Structured Query)
|
||||
* **場景**: "找出所有 **Cary Grant (Who)** 在 **車上 (Where)** 的片段"。
|
||||
* **邏輯**:
|
||||
1. 過濾 `analysis_5w1h` JSONB 欄位。
|
||||
2. `who` 包含 "Cary Grant" **AND** `where` 包含 "car"。
|
||||
3. 這種查詢比傳統關鍵字搜索更精準,因為它是經過 LLM 理解後的結構化數據。
|
||||
|
||||
### 3.3 動機與原因搜尋 (Why/How)
|
||||
* **場景**: "他為什麼要偷東西?"
|
||||
* **邏輯**:
|
||||
1. 針對 `analysis_5w1h.why` 進行語意比對。
|
||||
|
||||
---
|
||||
|
||||
## 4. 處理流程 (LLM Pipeline)
|
||||
|
||||
Rule 4 的生成需要呼叫 `llm_engine` (Gemma4) 服務。
|
||||
|
||||
### 4.1 演算法邏輯 (Pseudocode)
|
||||
|
||||
```python
|
||||
# 輸入: rule3_chunks (List of Scene Chunks)
|
||||
|
||||
# 1. 分組 (每 5-10 個場景一組)
|
||||
for group in chunks(rule3_chunks, size=7):
|
||||
|
||||
# 2. 準備 LLM 上下文
|
||||
context_text = "\n".join([chunk.summary for chunk in group])
|
||||
context_objects = aggregate_objects(group)
|
||||
|
||||
prompt = f"""
|
||||
Analyze the following video scenes and extract the 5W1H information.
|
||||
Scenes:
|
||||
{context_text}
|
||||
|
||||
Return JSON format:
|
||||
{{
|
||||
"summary": "A brief summary of these scenes.",
|
||||
"5w1h": {{
|
||||
"who": ["List of characters"],
|
||||
"what": ["Main events"],
|
||||
...
|
||||
}}
|
||||
}}
|
||||
"""
|
||||
|
||||
# 3. 呼叫 LLM (Gemma4 via Service Registry)
|
||||
response = llm_service.chat(prompt)
|
||||
result = parse_json(response)
|
||||
|
||||
# 4. 建立 Rule 4 Chunk
|
||||
rule4_chunk = {
|
||||
"summary": result["summary"],
|
||||
"analysis_5w1h": result["5w1h"],
|
||||
"start_frame": group[0].start_frame,
|
||||
"end_frame": group[-1].end_frame,
|
||||
"faces": aggregate_faces(group),
|
||||
"objects": aggregate_objects(group)
|
||||
}
|
||||
|
||||
# 5. 儲存並關聯
|
||||
rule4_id = store_rule4_chunk(rule4_chunk)
|
||||
for chunk in group:
|
||||
link_rule3_to_rule4(chunk.id, rule4_id)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 總結
|
||||
|
||||
Rule 4 將 Momentry 從「影片搜尋引擎」提升為**「影片知識圖譜」**。
|
||||
|
||||
| 特性 | 實作方式 |
|
||||
|------|----------|
|
||||
| **粒度** | 情節/敘事區塊 (5-10 場景) |
|
||||
| **核心技術** | LLM 5W1H 提取 (Gemma4) |
|
||||
| **數據結構** | 摘要文本 + JSONB 5W1H 結構 |
|
||||
| **向量內容** | 混合向量 (Summary + 5W1H) |
|
||||
| **適用場景** | 問答系統 (RAG)、劇情回顧、複雜條件過濾 |
|
||||
|
||||
**四層架構總覽:**
|
||||
1. **Rule 1 (Sentence)**: 精確台詞檢索。
|
||||
2. **Rule 2 (Visual)**: 畫面物件檢索。
|
||||
3. **Rule 3 (Scene)**: 場景上下文檢索。
|
||||
4. **Rule 4 (Summary)**: 劇情理解與知識問答。
|
||||
166
docs_v1.0/AI_AGENTS/TRANSLATION/TEXT_TRANSLATION.md
Normal file
166
docs_v1.0/AI_AGENTS/TRANSLATION/TEXT_TRANSLATION.md
Normal file
@@ -0,0 +1,166 @@
|
||||
# 翻譯 Agent (Translation Agent) 設計文件
|
||||
|
||||
| 項目 | 內容 |
|
||||
|------|------|
|
||||
| 建立者 | OpenCode |
|
||||
| 建立時間 | 2026-04-25 |
|
||||
| 文件版本 | V1.0 |
|
||||
| 用途 | 提供多語言文本翻譯服務 (應用於 Portal Chunk Detail) |
|
||||
|
||||
---
|
||||
|
||||
## 1. Agent 概覽
|
||||
|
||||
Translation Agent 負責將系統中的非結構化文本(如 Chunk 內容、摘要、5W1H 推論結果)翻譯為使用者指定的語言。
|
||||
在 Portal 的 **Chunk Search Detail** 頁面,當使用者瀏覽不同語言的影片內容時,此 Agent 提供即時翻譯支援。
|
||||
|
||||
### 1.1 資源註冊資訊 (Resource Registry)
|
||||
|
||||
當 Agent 啟動時,將向 **Resource Registry** 註冊以下資訊:
|
||||
|
||||
```json
|
||||
{
|
||||
"resource_id": "agent_text_translation_v1",
|
||||
"resource_type": "agent",
|
||||
"capabilities": ["translate_text", "detect_language", "batch_translate"],
|
||||
"category": "text_processing",
|
||||
"config": {
|
||||
"default_model": "gpt-4o-mini",
|
||||
"fallback_model": "local-llama-3-8b",
|
||||
"max_tokens": 4096,
|
||||
"supported_languages": ["zh-TW", "en-US", "ja-JP", "ko-KR"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心設計
|
||||
|
||||
### 2.1 輸入格式 (Input)
|
||||
|
||||
Agent 接收來自 Portal 或內部 API 的 JSON 請求:
|
||||
|
||||
```json
|
||||
{
|
||||
"text": "He walked into the room and saw a large red car.",
|
||||
"target_language": "zh-TW",
|
||||
"source_language": "auto",
|
||||
"context": {
|
||||
"domain": "movie_subtitle",
|
||||
"glossary": {
|
||||
"red car": "紅色跑車"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `text`: 待翻譯文本。
|
||||
- `target_language`: 目標語言 (BCP 47 格式)。
|
||||
- `context` (可選): 提供領域資訊或專有名詞對照表 (Glossary) 以提高準確度。
|
||||
|
||||
### 2.2 輸出格式 (Output)
|
||||
|
||||
Agent 回傳標準化 JSON:
|
||||
|
||||
```json
|
||||
{
|
||||
"translated_text": "他走進房間,看到一輛紅色跑車。",
|
||||
"source_language_detected": "en-US",
|
||||
"confidence": 0.98,
|
||||
"usage": {
|
||||
"input_tokens": 12,
|
||||
"output_tokens": 15
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Prompt 設計 (System Prompt)
|
||||
|
||||
為了確保翻譯風格符合 Momentry Core 的專業性(如準確的影視術語),我們使用以下 System Prompt:
|
||||
|
||||
```text
|
||||
You are a professional translator for Momentry Core, a digital asset management system specializing in video analysis.
|
||||
|
||||
## Guidelines:
|
||||
1. **Accuracy**: Translate the meaning accurately, maintaining the original tone.
|
||||
2. **Context Awareness**: If a glossary is provided in the context, strictly follow it.
|
||||
3. **Style**:
|
||||
- For subtitles: Keep it concise and natural for reading.
|
||||
- For technical terms (e.g., 5W1H, metadata): Use standard industry translations.
|
||||
4. **Format**: Preserve any JSON structure, markdown, or timestamps present in the input text. Do not translate code blocks.
|
||||
5. **Output**: Return ONLY the translated text in the requested format unless asked otherwise.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. API 端點設計
|
||||
|
||||
### 4.1 單一翻譯
|
||||
|
||||
```http
|
||||
POST /api/v1/agents/translate
|
||||
Content-Type: application/json
|
||||
X-Resource-Id: agent_text_translation_v1
|
||||
|
||||
{
|
||||
"text": "...",
|
||||
"target_language": "zh-TW"
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 批次翻譯 (Batch Translation)
|
||||
|
||||
針對 Chunk Detail 頁面可能一次顯示多個段落,支援批次翻譯:
|
||||
|
||||
```http
|
||||
POST /api/v1/agents/translate/batch
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"items": [
|
||||
{ "id": "chunk_001", "text": "..." },
|
||||
{ "id": "chunk_002", "text": "..." }
|
||||
],
|
||||
"target_language": "zh-TW"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 錯誤處理與容錯
|
||||
|
||||
- **模型降級 (Fallback)**: 若 `gpt-4o-mini` 超時或不可用,自動切換至本地模型 `local-llama-3-8b`。
|
||||
- **Token 超長**: 若文本超過 `max_tokens`,自動進行分段翻譯 (Split & Translate)。
|
||||
- **無效語言**: 若 `target_language` 不在支援列表中,回傳 `400 Bad Request`。
|
||||
|
||||
---
|
||||
|
||||
## 6. Portal 整合範例 (Chunk Detail)
|
||||
|
||||
在 Portal 的 `ChunkDetailView.vue` 中,翻譯功能的調用流程如下:
|
||||
|
||||
1. 使用者點擊「翻譯為 繁體中文」按鈕。
|
||||
2. Portal 發送 POST 請求至 `/api/v1/agents/translate`。
|
||||
3. 取得結果後,在不重新整理頁面的情況下更新 UI (顯示 `translated_text`)。
|
||||
|
||||
```typescript
|
||||
// Portal 前端調用範例
|
||||
async function translateChunkText(text: string, targetLang: string) {
|
||||
const response = await fetch('/api/v1/agents/translate', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ text, target_language: targetLang })
|
||||
});
|
||||
return response.json();
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 版本資訊
|
||||
|
||||
- 版本: V1.0
|
||||
- 建立日期: 2026-04-25
|
||||
Reference in New Issue
Block a user