feat: backup architecture docs, source code, and scripts
This commit is contained in:
Warren
337
docs_v1.0/CHUNKING/RULES/SCENE_BASED/CHUNK_RULE_3_COMPOSITE.md
Normal file
337
docs_v1.0/CHUNKING/RULES/SCENE_BASED/CHUNK_RULE_3_COMPOSITE.md
Normal file
@@ -0,0 +1,337 @@
|
||||
---
|
||||
document_type: "reference_doc"
|
||||
service: "MOMENTRY_CORE"
|
||||
title: "Unknown"
|
||||
date: "2026-03-28"
|
||||
version: "V1.0"
|
||||
status: "active"
|
||||
owner: "Warren"
|
||||
created_by: "OpenCode"
|
||||
tags:
|
||||
- "unknown"
|
||||
ai_query_hints:
|
||||
- "查詢 Unknown 的內容"
|
||||
- "Unknown 的主要目的是什麼？"
|
||||
- "如何操作或實施 Unknown？"
|
||||
---
|
||||
|
||||
---
|
||||
title: Chunk Rule 3 - Composite
|
||||
description: 父子關係 + frame_objects
|
||||
version: 1.0
|
||||
created: 2026-03-28
|
||||
updated: 2026-03-28
|
||||
service: MOMENTRY_CORE
|
||||
topic: chunk_rule
|
||||
document_type: spec
|
||||
rule_id: 3
|
||||
rule_name: Composite
|
||||
collection: momentry_rule3
|
||||
confidence_threshold: 0.8
|
||||
ai_agent_friendly: true
|
||||
---
|
||||
|
||||
# Chunk Rule 3 - Composite
|
||||
|
||||
| 項目 | 內容 |
|
||||
|------|------|
|
||||
| 建立者 | OpenCode |
|
||||
| 建立時間 | 2026-03-28 |
|
||||
| 文件版本 | V1.0 |
|
||||
| Rule ID | 3 |
|
||||
| Rule 名稱 | Composite |
|
||||
| Collection | `momentry_rule3` |
|
||||
| Confidence Threshold | > 0.8 |
|
||||
|
||||
---
|
||||
|
||||
## 版本歷史
|
||||
|
||||
| 版本 | 日期 | 目的 | 操作人 |
|
||||
|------|------|------|--------|
|
||||
| V1.0 | 2026-03-28 | 創建 Rule 3 規範 | OpenCode |
|
||||
|
||||
---
|
||||
|
||||
## 概述
|
||||
|
||||
Rule 3 (Composite) 是最完整的 Chunk 向量入庫規則。包含父子層級關係、frame_objects，以及所有可用資訊。
|
||||
|
||||
---
|
||||
|
||||
## 設計原則
|
||||
|
||||
### 輸入
|
||||
|
||||
- pre_chunk（來自 ASR/Cut/TimeBased 的原始分段）
|
||||
- frames（chunk 時間範圍內的所有 frames）
|
||||
- parent_chunk / child_chunks（層級關係）
|
||||
|
||||
### 處理
|
||||
|
||||
1. 同 Rule 2：收集 frame_objects (conf > 0.8)
|
||||
2. 建立父子層級關係
|
||||
3. 存入完整資訊
|
||||
|
||||
### 輸出
|
||||
|
||||
- chunk + frame_objects + parent_chunk_id + child_chunk_ids
|
||||
|
||||
---
|
||||
|
||||
## Collection 定義
|
||||
|
||||
### 建立 Collection
|
||||
|
||||
```bash
|
||||
curl -X PUT "http://localhost:6333/collections/momentry_rule3" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: <API_KEY>" \
|
||||
-d '{
|
||||
"vectors": {
|
||||
"size": 768,
|
||||
"distance": "Cosine"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Collection 參數
|
||||
|
||||
| 參數 | 值 |
|
||||
|------|-----|
|
||||
| Name | `momentry_rule3` |
|
||||
| Vector Size | 768 |
|
||||
| Distance | Cosine |
|
||||
| HNSW | m=16, ef_construct=100 |
|
||||
|
||||
---
|
||||
|
||||
## Payload 結構
|
||||
|
||||
### 欄位定義
|
||||
|
||||
| 欄位 | 類型 | 必填 | 說明 |
|
||||
|------|------|------|------|
|
||||
| `uuid` | String | ✅ | 影片 UUID (16 字元) |
|
||||
| `chunk_id` | String | ✅ | Chunk 唯一 ID |
|
||||
| `chunk_type` | String | ✅ | 類型：sentence/cut/time_based |
|
||||
| `chunk_index` | u32 | ✅ | Chunk 索引 (從 0 開始) |
|
||||
| `start_frame` | i64 | ✅ | 開始幀編號 |
|
||||
| `end_frame` | i64 | ✅ | 結束幀編號 |
|
||||
| `fps` | f64 | ✅ | 幀率 |
|
||||
| `original_text` | String | ✅ | 產生 vector 的原始文字 (ASR) |
|
||||
| `frame_objects` | String | ✅ | 涵蓋 frames 的物件描述 (conf > 0.8) |
|
||||
| `parent_chunk_id` | Option<String> | - | 父 Chunk ID |
|
||||
| `child_chunk_ids` | Vec<String> | - | 子 Chunk IDs |
|
||||
|
||||
### JSON 範例
|
||||
|
||||
```json
|
||||
{
|
||||
"uuid": "1636719dc31f78ac",
|
||||
"chunk_id": "sentence_parent_0001",
|
||||
"chunk_type": "sentence",
|
||||
"chunk_index": 1,
|
||||
"start_frame": 0,
|
||||
"end_frame": 2400,
|
||||
"fps": 24.0,
|
||||
"original_text": "Chapter 1: Introduction to the topic",
|
||||
"frame_objects": "person:5, car:2, building:3",
|
||||
"parent_chunk_id": null,
|
||||
"child_chunk_ids": ["sentence_0001", "sentence_0002", "sentence_0003"]
|
||||
}
|
||||
```
|
||||
|
||||
### Rust 結構
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct VectorPayloadRule3 {
|
||||
pub uuid: String,
|
||||
pub chunk_id: String,
|
||||
pub chunk_type: String,
|
||||
pub chunk_index: u32,
|
||||
pub start_frame: i64,
|
||||
pub end_frame: i64,
|
||||
pub fps: f64,
|
||||
pub original_text: String,
|
||||
pub frame_objects: String,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub parent_chunk_id: Option<String>,
|
||||
pub child_chunk_ids: Vec<String>,
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 父子層級關係
|
||||
|
||||
### 層級結構
|
||||
|
||||
```
|
||||
Parent Chunk (Story/Caption)
|
||||
|
|
||||
+-- Child Chunk 1 (sentence/cut/time_based)
|
||||
|
|
||||
+-- Child Chunk 2 (sentence/cut/time_based)
|
||||
|
|
||||
+-- Child Chunk 3 (sentence/cut/time_based)
|
||||
```
|
||||
|
||||
### chunk_id 命名規範
|
||||
|
||||
| 類型 | chunk_id 格式 | 範例 |
|
||||
|------|--------------|------|
|
||||
| Parent | `story_XXXX` | `story_0001` |
|
||||
| Parent | `caption_XXXX` | `caption_0001` |
|
||||
| Child | `sentence_XXXX` | `sentence_0001` |
|
||||
| Child | `cut_XXXX` | `cut_0001` |
|
||||
| Child | `time_based_XXXX` | `time_based_0001` |
|
||||
|
||||
### 範例
|
||||
|
||||
```json
|
||||
{
|
||||
"chunk_id": "story_0001",
|
||||
"chunk_type": "story",
|
||||
"parent_chunk_id": null,
|
||||
"child_chunk_ids": ["sentence_0001", "sentence_0002", "sentence_0003"]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## frame_objects 生成規則
|
||||
|
||||
（同 Rule 2，請參閱 CHUNK_RULE_2_FRAME_OBJECTS.md）
|
||||
|
||||
### 處理邏輯
|
||||
|
||||
1. 找出 chunk 時間範圍內的所有 frames
|
||||
2. 收集每個 frame 的物件識別結果（YOLO/Face/Pose）
|
||||
3. 過濾 confidence > 0.8 的物件
|
||||
4. 聚合物件名稱和數量
|
||||
|
||||
### 輸出字串
|
||||
|
||||
```
|
||||
"person:3, car:1, dog:2"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 時間計算
|
||||
|
||||
### Frame 轉時間
|
||||
|
||||
```
|
||||
start_time = start_frame / fps
|
||||
end_time = end_frame / fps
|
||||
```
|
||||
|
||||
### 範例
|
||||
|
||||
```
|
||||
- fps = 24.0
|
||||
- start_frame = 0
|
||||
- end_frame = 2400
|
||||
- start_time = 0 / 24.0 = 0 秒
|
||||
- end_time = 2400 / 24.0 = 100 秒
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 搜尋範例
|
||||
|
||||
### 語義搜尋（完整資訊）
|
||||
|
||||
```rust
|
||||
let query_vector = embed_text("找出有人在開車的相關場景").await?;
|
||||
let results = qdrant.search(
|
||||
"momentry_rule3",
|
||||
&query_vector,
|
||||
10,
|
||||
None
|
||||
).await?;
|
||||
```
|
||||
|
||||
### 父子層級搜尋
|
||||
|
||||
```bash
|
||||
# 搜尋 parent chunk 並取得所有 child chunks
|
||||
curl -X POST "http://localhost:6333/collections/momentry_rule3/points/search" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: <API_KEY>" \
|
||||
-d '{
|
||||
"vector": [0.123, -0.456, ...],
|
||||
"limit": 10,
|
||||
"with_payload": true,
|
||||
"filter": {
|
||||
"must": [
|
||||
{"key": "chunk_type", "match": {"value": "story"}}
|
||||
]
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### 根據 Child Chunk 找 Parent
|
||||
|
||||
```bash
|
||||
curl -X POST "http://localhost:6333/collections/momentry_rule3/points/search" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: <API_KEY>" \
|
||||
-d '{
|
||||
"vector": [0.123, -0.456, ...],
|
||||
"limit": 10,
|
||||
"with_payload": true,
|
||||
"filter": {
|
||||
"must": [
|
||||
{"key": "child_chunk_ids", "match": {"value": "sentence_0001"}}
|
||||
]
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 使用場景
|
||||
|
||||
| 場景 | 適用 |
|
||||
|------|------|
|
||||
| 需要父子層級關係搜尋 | ✅ 最佳 |
|
||||
| 需要完整資訊（ASR + 物件 + 層級） | ✅ 最佳 |
|
||||
| 跨層級分析 | ✅ 最佳 |
|
||||
| 僅 ASR 搜尋 | ❌ 請用 Rule 1 |
|
||||
| 僅需物件輔助搜尋 | ❌ 請用 Rule 2 |
|
||||
|
||||
---
|
||||
|
||||
## 優點與限制
|
||||
|
||||
### 優點
|
||||
|
||||
- 最完整的資訊
|
||||
- 支援父子層級搜尋
|
||||
- 可進行跨層級分析
|
||||
|
||||
### 限制
|
||||
|
||||
- 資料量最大
|
||||
- 搜尋速度相對較慢
|
||||
- 實作複雜度最高
|
||||
|
||||
---
|
||||
|
||||
## 相關文件
|
||||
|
||||
| 文件 | 用途 |
|
||||
|------|------|
|
||||
| CHUNK_RULES_SPEC.md | 規則總覽 |
|
||||
| CHUNK_RULE_1_SIMPLE.md | Rule 1 規範 |
|
||||
| CHUNK_RULE_2_FRAME_OBJECTS.md | Rule 2 規範 |
|
||||
| CHUNK_SPEC.md | Chunk 基礎規範 |
|
||||
| CHUNK_DESIGN.md | Chunk 設計架構 |
|
||||
|
||||
---
|
||||
|
||||
**文件結束**
|
||||
215
docs_v1.0/CHUNKING/RULES/SCENE_BASED/CHUNK_RULE_3_SCENE.md
Normal file
215
docs_v1.0/CHUNKING/RULES/SCENE_BASED/CHUNK_RULE_3_SCENE.md
Normal file
@@ -0,0 +1,215 @@
|
||||
---
|
||||
document_type: "architecture_design"
|
||||
service: "MOMENTRY_CORE"
|
||||
title: "Momentry Core Chunk Rule 3: 場景聚合級檢索 (Scene Composite Chunk) (v1.0)"
|
||||
date: "2026-04-21"
|
||||
version: "V1.0"
|
||||
status: "active"
|
||||
owner: "Warren"
|
||||
created_by: "OpenCode"
|
||||
tags:
|
||||
- "momentry"
|
||||
- "core"
|
||||
- "rule"
|
||||
- "chunk"
|
||||
ai_query_hints:
|
||||
- "查詢 Momentry Core Chunk Rule 3: 場景聚合級檢索 (Scene Composite Chunk) (v1.0) 的內容"
|
||||
- "Momentry Core Chunk Rule 3: 場景聚合級檢索 (Scene Composite Chunk) (v1.0) 的主要目的是什麼？"
|
||||
- "如何操作或實施 Momentry Core Chunk Rule 3: 場景聚合級檢索 (Scene Composite Chunk) (v1.0)？"
|
||||
---
|
||||
|
||||
# Momentry Core Chunk Rule 3: 場景聚合級檢索 (Scene Composite Chunk) (v1.0)
|
||||
|
||||
| 項目 | 內容 |
|
||||
|------|------|
|
||||
| 建立者 | OpenCode |
|
||||
| 建立時間 | 2026-04-21 |
|
||||
| 文件版本 | V1.0 |
|
||||
|
||||
---
|
||||
|
||||
## 版本歷史
|
||||
|
||||
| 版本 | 日期 | 目的 | 操作人 | 工具/模型 |
|
||||
|------|------|------|--------|-----------|
|
||||
| V1.0 | 2026-04-21 | 定義 Rule 3: 基於 Cut 點的場景級父子聚合結構 | OpenCode | OpenCode / Qwen3.6-Plus |
|
||||
|
||||
---
|
||||
|
||||
## 0. 設計目標
|
||||
|
||||
**Rule 3** 的核心概念是**「場景理解」(Scene Understanding)**。利用 **Cut Processor** 偵測到的鏡頭切換點，將影片劃分為語意完整的場景區塊，並聚合內部的所有句子與視覺資訊。
|
||||
|
||||
- **核心原則**: 一個鏡頭/場景 (Cut) = 一個 Parent Chunk。
|
||||
- **結構**: 採用 **Parent-Child (父子)** 架構。
|
||||
- **Parent (Rule 3)**: 代表整個場景，包含摘要 (Summary) 與聚合向量。
|
||||
- **Children (Rule 1/2)**: 場景內包含的具體句子與視覺幀。
|
||||
- **優勢**: 支援跨句子的長語境搜尋 (例如搜尋整個情節的摘要，而非單一單詞)。
|
||||
|
||||
---
|
||||
|
||||
## 1. 數據源與聚合邏輯
|
||||
|
||||
Rule 3 不直接從原始影片產生，而是依賴 **Rule 1** 與 **Rule 2** 的產出。
|
||||
|
||||
1. **Cut Processor (Primary)**: 提供場景的邊界。
|
||||
- *定義*: `start_frame`, `end_frame` 為一個完整鏡頭。
|
||||
2. **Rule 1 Chunks (Children)**: 收集該場景內所有的 ASR 語句 (Sentences)。
|
||||
3. **Rule 2 Chunks (Children)**: 收集該場景內所有的視覺幀數據 (Visual Frames)。
|
||||
4. **Summary Generation**:
|
||||
- 為了讓 Parent Chunk 具備搜尋能力，系統會將所有子 Chunk 的內容 (ASR 文本 + 物件標籤) 組合成一段「場景描述」，並由 LLM (選用) 或規則生成一段 **Summary**。
|
||||
|
||||
---
|
||||
|
||||
## 2. Chunk 結構定義
|
||||
|
||||
### 2.1 資料庫結構 (PostgreSQL)
|
||||
|
||||
採用 **Parent-Child** 設計，Rule 3 為 Parent，Rule 1/2 透過 `parent_id` 指向 Rule 3。
|
||||
|
||||
```sql
|
||||
-- Parent Table (Rule 3: Scenes)
|
||||
CREATE TABLE parent_chunks (
|
||||
id UUID PRIMARY KEY,
|
||||
asset_uuid UUID NOT NULL,
|
||||
chunk_type VARCHAR(20) DEFAULT 'scene',
|
||||
|
||||
-- 時間軸 (幀為權威)
|
||||
start_frame INT NOT NULL,
|
||||
end_frame INT NOT NULL,
|
||||
start_time_sec DOUBLE PRECISION,
|
||||
end_time_sec DOUBLE PRECISION,
|
||||
|
||||
-- 場景內容 (用於向量索引)
|
||||
summary TEXT NOT NULL, -- 場景摘要 (由內部 ASR 聚合而成)
|
||||
|
||||
-- 元數據聚合
|
||||
faces JSONB, -- 場景內所有出現過的人物 ID
|
||||
speakers JSONB, -- 場景內所有出現過的說話者 ID
|
||||
objects JSONB, -- 場景內出現過的高信心物件
|
||||
|
||||
-- 向量索引
|
||||
embedding vector(768), -- 摘要的向量
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Child Tables (Rule 1/2) 增加 parent_id
|
||||
-- (需對 chunks_rule1 與 chunks_rule2 執行 ALTER TABLE 增加欄位)
|
||||
ALTER TABLE chunks_rule1 ADD COLUMN parent_id UUID REFERENCES parent_chunks(id);
|
||||
ALTER TABLE chunks_rule2 ADD COLUMN parent_id UUID REFERENCES parent_chunks(id);
|
||||
```
|
||||
|
||||
### 2.2 JSON 產出範例 (嵌套結構)
|
||||
|
||||
Rule 3 的 API 返回應包含聚合後的子項目。
|
||||
|
||||
```json
|
||||
{
|
||||
"chunk_id": "550e...0003",
|
||||
"type": "scene",
|
||||
"summary": "Peter Joshua 和 Regina 在車上討論關於金錢的危機。畫面顯示兩人在車內，背景為夜間街道。",
|
||||
"start_frame": 1000,
|
||||
"end_frame": 1500,
|
||||
"children": [
|
||||
{
|
||||
"type": "sentence",
|
||||
"content": "我們必須在那之前找到那筆錢。",
|
||||
"speaker": "SPEAKER_00",
|
||||
"start_frame": 1100,
|
||||
"end_frame": 1200
|
||||
},
|
||||
{
|
||||
"type": "visual_frame",
|
||||
"content": "car, person, night, street",
|
||||
"frame_objects": [ ... ]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"faces": ["cary_grant", "audrey_hepburn"],
|
||||
"speakers": ["SPEAKER_00", "SPEAKER_01"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 搜尋能力定義
|
||||
|
||||
Rule 3 專為**宏觀理解**與**摘要檢索**設計。
|
||||
|
||||
### 3.1 場景摘要搜尋 (Summary Search)
|
||||
* **場景**: "尋找他們討論分贓的場景" (可能包含多句對話)。
|
||||
* **邏輯**:
|
||||
1. Query: "Discussion about splitting the money".
|
||||
2. Match: 搜尋 `parent_chunks.summary` 的向量。
|
||||
3. 結果：直接返回整個場景 (Parent)，而非零碎的句子。
|
||||
|
||||
### 3.2 混合檢索 (Hybrid Retrieval)
|
||||
* **場景**: 使用者搜尋 "槍戰"。
|
||||
* **策略**:
|
||||
1. **Hit**: Rule 2 (Visual) 命中 (偵測到 "gun")。
|
||||
2. **Expand**: 系統自動向上查找該 Rule 2 所屬的 Rule 3 Parent。
|
||||
3. **Return**: 返回該場面的完整上下文 (包含槍戰前後的對話)。
|
||||
|
||||
---
|
||||
|
||||
## 4. 處理流程 (Aggregation Pipeline)
|
||||
|
||||
Rule 3 是在 Rule 1 與 Rule 2 完成後執行的「後處理」步驟。
|
||||
|
||||
### 4.1 演算法邏輯 (Pseudocode)
|
||||
|
||||
```python
|
||||
# 輸入: cuts (List of boundaries), rule1_chunks, rule2_chunks
|
||||
|
||||
for cut in cuts:
|
||||
scene_start = cut.start_frame
|
||||
scene_end = cut.end_frame
|
||||
|
||||
# 1. 收集子元素 (Children)
|
||||
children_sentences = get_children_in_range(scene_start, scene_end, rule1_chunks)
|
||||
children_visuals = get_children_in_range(scene_start, scene_end, rule2_chunks)
|
||||
|
||||
# 2. 聚合元數據
|
||||
scene_faces = aggregate_unique_ids(children_sentences, "face_ids")
|
||||
scene_faces.update(aggregate_unique_ids(children_visuals, "face_ids"))
|
||||
|
||||
scene_speakers = aggregate_unique_ids(children_sentences, "speaker_id")
|
||||
|
||||
# 3. 生成 Summary
|
||||
# 組合所有 ASR 文本
|
||||
full_text = " ".join([c.content for c in children_sentences])
|
||||
# 組合所有視覺標籤
|
||||
visual_context = ", ".join(get_top_objects(children_visuals))
|
||||
|
||||
summary = f"[Scene] {full_text}. Visuals: {visual_context}."
|
||||
|
||||
# 4. 建立 Parent Chunk
|
||||
parent = {
|
||||
"start_frame": scene_start,
|
||||
"end_frame": scene_end,
|
||||
"summary": summary,
|
||||
"faces": list(scene_faces),
|
||||
"speakers": list(scene_speakers)
|
||||
}
|
||||
|
||||
# 5. 儲存 Parent，並將子元素關聯到此 Parent ID
|
||||
parent_id = store_parent_chunk(parent)
|
||||
link_children_to_parent(children_sentences + children_visuals, parent_id)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 總結
|
||||
|
||||
Rule 3 是 Momentry 檢索架構的**頂層視角**。
|
||||
|
||||
| 特性 | 實作方式 |
|
||||
|------|----------|
|
||||
| **粒度** | 場景/鏡頭 (Scene/Cut) |
|
||||
| **資料來源** | Rule 1 (Text) + Rule 2 (Visual) |
|
||||
| **核心內容** | 場景摘要 (Summary) + 聚合元數據 |
|
||||
| **向量內容** | 僅對 Summary 進行 Embedding，確保向量代表宏觀語意 |
|
||||
| **適用場景** | 尋找特定情節、理解長段落上下文、場景過濾 |
|
||||
|
||||
透過 Rule 1/2/3 的三層架構，系統能同時滿足**微觀精確檢索** (Rule 1) 與 **宏觀場景理解** (Rule 3) 的需求。
|
||||
202
docs_v1.0/CHUNKING/RULES/TEXT_BASED/CHUNK_RULE_1_SENTENCE.md
Normal file
202
docs_v1.0/CHUNKING/RULES/TEXT_BASED/CHUNK_RULE_1_SENTENCE.md
Normal file
@@ -0,0 +1,202 @@
|
||||
---
|
||||
document_type: "architecture_design"
|
||||
service: "MOMENTRY_CORE"
|
||||
title: "Momentry Core Chunk Rule 1: 句子級檢索 (Sentence Chunk) (v1.0)"
|
||||
date: "2026-04-21"
|
||||
version: "V1.0"
|
||||
status: "active"
|
||||
owner: "Warren"
|
||||
created_by: "OpenCode"
|
||||
tags:
|
||||
- "momentry"
|
||||
- "core"
|
||||
- "句子級檢索"
|
||||
- "rule"
|
||||
ai_query_hints:
|
||||
- "查詢 Momentry Core Chunk Rule 1: 句子級檢索 (Sentence Chunk) (v1.0) 的內容"
|
||||
- "Momentry Core Chunk Rule 1: 句子級檢索 (Sentence Chunk) (v1.0) 的主要目的是什麼？"
|
||||
- "如何操作或實施 Momentry Core Chunk Rule 1: 句子級檢索 (Sentence Chunk) (v1.0)？"
|
||||
---
|
||||
|
||||
# Momentry Core Chunk Rule 1: 句子級檢索 (Sentence Chunk) (v1.0)
|
||||
|
||||
| 項目 | 內容 |
|
||||
|------|------|
|
||||
| 建立者 | OpenCode |
|
||||
| 建立時間 | 2026-04-21 |
|
||||
| 文件版本 | V1.0 |
|
||||
|
||||
---
|
||||
|
||||
## 版本歷史
|
||||
|
||||
| 版本 | 日期 | 目的 | 操作人 | 工具/模型 |
|
||||
|------|------|------|--------|-----------|
|
||||
| V1.0 | 2026-04-21 | 定義 Rule 1: 單一語句為 Chunk 的數據結構與搜尋邏輯 | OpenCode | OpenCode / Qwen3.6-Plus |
|
||||
|
||||
---
|
||||
|
||||
## 0. 設計目標
|
||||
|
||||
**Rule 1** 的核心概念是**「原子化語義」**。將影片內容切分到「一句話」的粒度，以便進行最高精度的語意搜尋與元數據過濾。
|
||||
|
||||
- **核心原則**: One Sentence = One Chunk。
|
||||
- **時間權威**: 基於 `frame_number` (由 `ffprobe` 定義的 FPS 計算)。
|
||||
- **多模態關聯**: 每個句子 Chunk 必須攜帶該時間區間內的 **Speaker (說話者)** 與 **Faces (出現人物)** 資訊。
|
||||
|
||||
---
|
||||
|
||||
## 1. 數據源與聚合邏輯
|
||||
|
||||
Rule 1 的生成依賴三個上游處理器的產出：
|
||||
|
||||
1. **ASR (Primary)**: 提供文本內容 (`text`)、起始時間 (`start_time`)、結束時間 (`end_time`)。
|
||||
2. **ASRX (Speaker)**: 提供說話者 ID (`speaker_id`)。
|
||||
- *聚合策略*: 使用 ASR 的時間區間去對齊 ASRX，取該區間內**佔比最高**的 `speaker_id`。
|
||||
3. **Face (Visual)**: 提供幀級別的人物 ID (`face_id`)。
|
||||
- *聚合策略*: 在 ASR 的 `[start_frame, end_frame]` 區間內，收集所有出現的 `face_id`。若同一 ID 出現多次，去重後形成 `face_ids` 陣列。
|
||||
|
||||
---
|
||||
|
||||
## 2. Chunk 結構定義
|
||||
|
||||
### 2.1 資料庫結構 (PostgreSQL)
|
||||
|
||||
```sql
|
||||
CREATE TABLE chunks_rule1 (
|
||||
id UUID PRIMARY KEY,
|
||||
asset_uuid UUID NOT NULL, -- 關聯 assets 表
|
||||
chunk_type VARCHAR(20) DEFAULT 'sentence', -- 對應 ChunkType::Sentence
|
||||
|
||||
-- 時間軸 (幀為權威)
|
||||
start_frame INT NOT NULL,
|
||||
end_frame INT NOT NULL,
|
||||
start_time_sec DOUBLE PRECISION, -- 參考值: start_frame / fps
|
||||
end_time_sec DOUBLE PRECISION, -- 參考值: end_frame / fps
|
||||
fps DOUBLE PRECISION NOT NULL,
|
||||
|
||||
-- 核心內容
|
||||
content TEXT NOT NULL, -- ASR 識別出的文字
|
||||
|
||||
-- 關聯元數據 (Metadata)
|
||||
speaker_id VARCHAR(50), -- ASRX 產出
|
||||
face_ids JSONB, -- Face 產出，例如 ["face_01", "face_02"]
|
||||
|
||||
-- 向量與索引
|
||||
embedding vector(768), -- nomic-embed-text-v2-moe
|
||||
search_vector tsvector, -- PostgreSQL BM25
|
||||
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
```
|
||||
|
||||
### 2.2 JSON 產出範例 (供前端 API 返回)
|
||||
|
||||
當 API 搜尋 Rule 1 時，返回結構如下：
|
||||
|
||||
```json
|
||||
{
|
||||
"chunk_id": "550e...0001",
|
||||
"type": "sentence",
|
||||
"content": "這是一個關於尋找真相的故事。",
|
||||
"start_frame": 1200,
|
||||
"end_frame": 1250,
|
||||
"start_time_sec": 40.04,
|
||||
"end_time_sec": 41.71,
|
||||
"metadata": {
|
||||
"speaker": "SPEAKER_00",
|
||||
"faces": [
|
||||
{ "face_id": "person_a", "confidence": 0.98 }
|
||||
]
|
||||
},
|
||||
"highlight": "這是一個關於<span class='highlight'>尋找真相</span>的故事。"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 搜尋能力定義
|
||||
|
||||
Rule 1 支援三種主要搜尋模式：
|
||||
|
||||
### 3.1 語意搜尋 (Vector Search)
|
||||
* **場景**: "有人提到錢嗎？" (即使影片沒說 "錢"，而是說 "鈔票" 也能搜到)。
|
||||
* **邏輯**:
|
||||
1. 將 Query 透過 Ollama (`nomic-v2-moe`) 轉為 768-dim 向量。
|
||||
2. 在 Qdrant (`collection: momentry_rule1`) 中進行 Cosine 相似度比對。
|
||||
3. **Filter**: 可加入 `metadata.speaker == "SPEAKER_00"`。
|
||||
|
||||
### 3.2 關鍵字搜尋 (BM25 Search)
|
||||
* **場景**: "搜尋確切字串 'Charade 1963'"。
|
||||
* **邏輯**:
|
||||
1. 使用 PostgreSQL `tsvector` 進行全文檢索。
|
||||
2. 適合精確匹配專有名詞。
|
||||
|
||||
### 3.3 過濾搜尋 (Faceted Search)
|
||||
* **場景**: "找出 **Audrey Hepburn (Face)** 說話的所有片段"。
|
||||
* **邏輯**:
|
||||
1. `face_ids` 包含 "Audrey Hepburn" 的 ID。
|
||||
2. `speaker_id` 不為空 (代表她在說話)。
|
||||
3. 檢索符合條件的 Chunks。
|
||||
|
||||
---
|
||||
|
||||
## 4. 處理流程 (Processing Pipeline)
|
||||
|
||||
### 4.1 聚合演算法 (Pseudocode)
|
||||
|
||||
```python
|
||||
# 輸入: asr_segments (List), asrx_segments (List), face_frames (List)
|
||||
# 常數: FPS (來自 ffprobe)
|
||||
|
||||
for seg in asr_segments:
|
||||
# 1. 確定時間範圍 (Frames)
|
||||
start_f = int(seg.start_time * FPS)
|
||||
end_f = int(seg.end_time * FPS)
|
||||
|
||||
# 2. 匹配 Speaker (取重疊時間最長的)
|
||||
speaker = find_majority_speaker(start_f, end_f, asrx_segments)
|
||||
|
||||
# 3. 聚合 Faces (收集區間內出現過的所有唯一 ID)
|
||||
faces = get_unique_faces(start_f, end_f, face_frames)
|
||||
|
||||
# 4. 建立 Chunk
|
||||
chunk = {
|
||||
"content": seg.text,
|
||||
"start_frame": start_f,
|
||||
"end_frame": end_f,
|
||||
"speaker_id": speaker,
|
||||
"face_ids": faces
|
||||
}
|
||||
|
||||
# 5. 寫入 DB
|
||||
store_chunk_rule1(chunk)
|
||||
```
|
||||
|
||||
### 4.2 時間邊界處理
|
||||
|
||||
若 ASR 的 `end_time` 與 ASRX 的 `start_time` 有微小誤差 (例如 0.05s)，系統應容忍 **±2 frames** 的誤差範圍進行匹配。
|
||||
|
||||
---
|
||||
|
||||
## 5. 向量嵌入策略
|
||||
|
||||
* **嵌入模型**: `nomic-embed-text-v2-moe` (768-dim)。
|
||||
* **嵌入內容**: 僅使用 `content` (句子文字)。
|
||||
* *原因*: 避免 speaker 或 face 的 metadata 干擾語意向量空間，確保語意純淨。Metadata 僅用於過濾 (Filter)。
|
||||
|
||||
---
|
||||
|
||||
## 6. 總結
|
||||
|
||||
Rule 1 提供了**最細緻**的影片理解層級。
|
||||
|
||||
| 特性 | 實作方式 |
|
||||
|------|----------|
|
||||
| **粒度** | 句子 (Sentence) |
|
||||
| **時間精度** | Frame 級別 (由 FPS 換算) |
|
||||
| **人物標記** | 自動關聯 Face ID (Visual) |
|
||||
| **說話者標記** | 自動關聯 Speaker ID (Audio) |
|
||||
| **適用場景** | 尋找特定台詞、某人說了什麼、特定鏡頭對話 |
|
||||
|
||||
此規範確保了所有 Rule 1 Chunk 在進入資料庫前，都已經完成了多模態數據的融合 (Audio + Visual + Text)。
|
||||
378
docs_v1.0/CHUNKING/RULES/TEXT_BASED/CHUNK_RULE_1_SIMPLE.md
Normal file
378
docs_v1.0/CHUNKING/RULES/TEXT_BASED/CHUNK_RULE_1_SIMPLE.md
Normal file
@@ -0,0 +1,378 @@
|
||||
---
|
||||
document_type: "reference_doc"
|
||||
service: "MOMENTRY_CORE"
|
||||
title: "Unknown"
|
||||
date: "2026-03-28"
|
||||
version: "V1.0"
|
||||
status: "active"
|
||||
owner: "Warren"
|
||||
created_by: "OpenCode"
|
||||
tags:
|
||||
- "unknown"
|
||||
ai_query_hints:
|
||||
- "查詢 Unknown 的內容"
|
||||
- "Unknown 的主要目的是什麼？"
|
||||
- "如何操作或實施 Unknown？"
|
||||
---
|
||||
|
||||
---
|
||||
title: Chunk Rule 1 - Simple
|
||||
description: 直接轉換，無父子關係，無 frame objects
|
||||
version: 1.0
|
||||
created: 2026-03-28
|
||||
updated: 2026-03-28
|
||||
service: MOMENTRY_CORE
|
||||
topic: chunk_rule
|
||||
document_type: spec
|
||||
rule_id: 1
|
||||
rule_name: Simple
|
||||
collection: momentry_rule1
|
||||
ai_agent_friendly: true
|
||||
---
|
||||
|
||||
# Chunk Rule 1 - Simple
|
||||
|
||||
| 項目 | 內容 |
|
||||
|------|------|
|
||||
| 建立者 | OpenCode |
|
||||
| 建立時間 | 2026-03-28 |
|
||||
| 文件版本 | V1.0 |
|
||||
| Rule ID | 1 |
|
||||
| Rule 名稱 | Simple |
|
||||
| Collection | `momentry_rule1` |
|
||||
|
||||
---
|
||||
|
||||
## 版本歷史
|
||||
|
||||
| 版本 | 日期 | 目的 | 操作人 |
|
||||
|------|------|------|--------|
|
||||
| V1.0 | 2026-03-28 | 創建 Rule 1 規範 | OpenCode |
|
||||
|
||||
---
|
||||
|
||||
## 概述
|
||||
|
||||
Rule 1 (Simple) 是最基本的 Chunk 向量入庫規則。直接將 pre_chunk 轉換為 chunk，不包含父子關係和 frame objects。
|
||||
|
||||
---
|
||||
|
||||
## 設計原則
|
||||
|
||||
### 輸入
|
||||
|
||||
- pre_chunk（來自 ASR/Cut/TimeBased 的原始分段）
|
||||
|
||||
### 處理
|
||||
|
||||
- 直接轉換，無額外處理
|
||||
|
||||
### 輸出
|
||||
|
||||
- chunk（與 pre_chunk 邊界相同）
|
||||
|
||||
---
|
||||
|
||||
## Collection 定義
|
||||
|
||||
### 建立 Collection
|
||||
|
||||
```bash
|
||||
curl -X PUT "http://localhost:6333/collections/momentry_rule1" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: <API_KEY>" \
|
||||
-d '{
|
||||
"vectors": {
|
||||
"size": 768,
|
||||
"distance": "Cosine"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Collection 參數
|
||||
|
||||
| 參數 | 值 |
|
||||
|------|-----|
|
||||
| Name | `momentry_rule1` |
|
||||
| Vector Size | 768 |
|
||||
| Distance | Cosine |
|
||||
| HNSW | m=16, ef_construct=100 |
|
||||
|
||||
---
|
||||
|
||||
## 嵌入模型
|
||||
|
||||
### 專用模型
|
||||
Rule 1 專用 **`nomic-embed-text-v2-moe:latest`** 嵌入模型，提供完整多語言支持：
|
||||
|
||||
#### 模型特性
|
||||
| 特性 | 說明 |
|
||||
|------|------|
|
||||
| **模型名稱** | `nomic-embed-text-v2-moe:latest` |
|
||||
| **模型類型** | Mixture of Experts (MoE) 架構 |
|
||||
| **向量維度** | 768 維 |
|
||||
| **多語言支持** | ✅ 完整支持（英語、中文、日語、韓語等） |
|
||||
| **模型大小** | 475.29 MB |
|
||||
| **推理速度** | 快速，適合實時應用 |
|
||||
|
||||
#### 模型優勢
|
||||
1. **完整多語言能力**: 原生支持多語言文本嵌入，無需語言檢測
|
||||
2. **高效能架構**: MoE 架構提供高效推理
|
||||
3. **統一向量空間**: 所有語言共享相同的 768 維向量空間
|
||||
4. **Ollama 集成**: 通過標準 Ollama API 直接調用
|
||||
|
||||
### 模型配置
|
||||
```rust
|
||||
// Rust 代碼中使用
|
||||
let embedder = Embedder::new("nomic-embed-text-v2-moe:latest".to_string());
|
||||
let vector = embedder.embed_text("搜索文本").await?;
|
||||
```
|
||||
|
||||
```bash
|
||||
# 直接調用 Ollama API
|
||||
curl -X POST "http://localhost:11434/api/embeddings" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "nomic-embed-text-v2-moe:latest",
|
||||
"prompt": "需要嵌入的文本內容"
|
||||
}'
|
||||
```
|
||||
|
||||
### 多語言示例
|
||||
```rust
|
||||
// 英語文本
|
||||
let english_vector = embedder.embed_text("Hello world, this is a test").await?;
|
||||
|
||||
// 中文文本
|
||||
let chinese_vector = embedder.embed_text("你好世界，這是一個測試").await?;
|
||||
|
||||
// 日語文本
|
||||
let japanese_vector = embedder.embed_text("こんにちは世界、これはテストです").await?;
|
||||
|
||||
// 韓語文本
|
||||
let korean_vector = embedder.embed_text("안녕하세요 세계, 이것은 테스트입니다").await?;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Payload 結構
|
||||
|
||||
### 欄位定義
|
||||
|
||||
| 欄位 | 類型 | 必填 | 說明 |
|
||||
|------|------|------|------|
|
||||
| `uuid` | String | ✅ | 影片 UUID (16 字元) |
|
||||
| `chunk_id` | String | ✅ | Chunk 唯一 ID |
|
||||
| `chunk_type` | String | ✅ | 類型：sentence/cut/time_based |
|
||||
| `chunk_index` | u32 | ✅ | Chunk 索引 (從 0 開始) |
|
||||
| `start_frame` | i64 | ✅ | 開始幀編號 |
|
||||
| `end_frame` | i64 | ✅ | 結束幀編號 |
|
||||
| `fps` | f64 | ✅ | 幀率 |
|
||||
| `original_text` | String | ✅ | 產生 vector 的原始文字 (ASR) |
|
||||
|
||||
### JSON 範例
|
||||
|
||||
```json
|
||||
{
|
||||
"uuid": "1636719dc31f78ac",
|
||||
"chunk_id": "sentence_0001",
|
||||
"chunk_type": "sentence",
|
||||
"chunk_index": 1,
|
||||
"start_frame": 252,
|
||||
"end_frame": 378,
|
||||
"fps": 24.0,
|
||||
"original_text": "Hello world, this is a test message"
|
||||
}
|
||||
```
|
||||
|
||||
### Rust 結構
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct VectorPayloadRule1 {
|
||||
pub uuid: String,
|
||||
pub chunk_id: String,
|
||||
pub chunk_type: String,
|
||||
pub chunk_index: u32,
|
||||
pub start_frame: i64,
|
||||
pub end_frame: i64,
|
||||
pub fps: f64,
|
||||
pub original_text: String,
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 時間計算
|
||||
|
||||
### Frame 轉時間
|
||||
|
||||
```
|
||||
start_time = start_frame / fps
|
||||
end_time = end_frame / fps
|
||||
```
|
||||
|
||||
### 範例
|
||||
|
||||
```
|
||||
- fps = 24.0
|
||||
- start_frame = 252
|
||||
- end_frame = 378
|
||||
- start_time = 252 / 24.0 = 10.5 秒
|
||||
- end_time = 378 / 24.0 = 15.75 秒
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 搜尋範例
|
||||
|
||||
### 語義搜尋（使用 nomic-embed-text-v2-moe:latest）
|
||||
|
||||
```rust
|
||||
use crate::core::embedding::comic_embed::Embedder;
|
||||
|
||||
// 1. 初始化嵌入器
|
||||
let embedder = Embedder::new("nomic-embed-text-v2-moe:latest".to_string());
|
||||
|
||||
// 2. 生成查詢向量（支持多語言）
|
||||
let query_text = "找出有人在說話的片段"; // 中文查詢
|
||||
// let query_text = "Find segments where someone is speaking"; // 英文查詢
|
||||
// let query_text = "誰かが話しているセグメントを見つける"; // 日文查詢
|
||||
|
||||
let query_vector = embedder.embed_query(query_text).await?;
|
||||
|
||||
// 3. 在 Qdrant 中搜索
|
||||
let results = qdrant.search(
|
||||
"momentry_rule1",
|
||||
&query_vector,
|
||||
10,
|
||||
None
|
||||
).await?;
|
||||
|
||||
// 4. 處理結果
|
||||
for result in results {
|
||||
println!("Score: {}, Chunk ID: {}", result.score, result.payload["chunk_id"]);
|
||||
}
|
||||
```
|
||||
|
||||
### 批量嵌入示例
|
||||
|
||||
```rust
|
||||
// 批量嵌入多語言文本
|
||||
let texts = vec![
|
||||
"Hello world, this is English text",
|
||||
"你好世界，這是中文文本",
|
||||
"こんにちは世界、これは日本語のテキストです",
|
||||
"안녕하세요 세계, 이것은 한국어 텍스트입니다"
|
||||
];
|
||||
|
||||
let mut vectors = Vec::new();
|
||||
for text in texts {
|
||||
let vector = embedder.embed_document(text).await?;
|
||||
vectors.push(vector);
|
||||
}
|
||||
|
||||
// 批量存入 Qdrant
|
||||
for (i, vector) in vectors.iter().enumerate() {
|
||||
qdrant.upsert(
|
||||
"momentry_rule1",
|
||||
i as u64,
|
||||
vector,
|
||||
Some(json!({
|
||||
"uuid": "test_uuid",
|
||||
"chunk_id": format!("test_{}", i),
|
||||
"chunk_type": "sentence",
|
||||
"chunk_index": i as u32,
|
||||
"original_text": texts[i]
|
||||
}))
|
||||
).await?;
|
||||
}
|
||||
```
|
||||
|
||||
### 依賴 Qdrant Filter
|
||||
|
||||
```bash
|
||||
curl -X POST "http://localhost:6333/collections/momentry_rule1/points/search" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: <API_KEY>" \
|
||||
-d '{
|
||||
"vector": [0.123, -0.456, ...],
|
||||
"limit": 10,
|
||||
"with_payload": true,
|
||||
"filter": {
|
||||
"must": [
|
||||
{"key": "uuid", "match": {"value": "1636719dc31f78ac"}},
|
||||
{"key": "chunk_type", "match": {"value": "sentence"}}
|
||||
]
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 使用場景
|
||||
|
||||
### 多語言搜索場景
|
||||
| 場景 | 適用 | 多語言支持 |
|
||||
|------|------|------------|
|
||||
| 僅 ASR 文字語義搜尋 | ✅ 最佳 | ✅ 英語、中文、日語等 |
|
||||
| 簡單問答系統 | ✅ 最佳 | ✅ 跨語言問答 |
|
||||
| 基礎影片檢索 | ✅ 最佳 | ✅ 多語言檢索 |
|
||||
| 多語言內容分析 | ✅ 適合 | ✅ 混合語言內容 |
|
||||
| 跨語言相似度匹配 | ✅ 適合 | ✅ 語言無關嵌入 |
|
||||
| 需要物體辨識結果 | ❌ 請用 Rule 2/3 | - |
|
||||
| 需要父子層級關係 | ❌ 請用 Rule 3 | - |
|
||||
|
||||
### 多語言示例場景
|
||||
1. **中文搜索英文內容**: 用戶用中文查詢，找到英文影片片段
|
||||
2. **跨語言相似內容發現**: 不同語言描述相同概念的內容匹配
|
||||
3. **混合語言影片處理**: 影片中包含多種語言對話的場景
|
||||
4. **全球化內容檢索**: 支持多國用戶使用母語搜索
|
||||
|
||||
---
|
||||
|
||||
## 優點與限制
|
||||
|
||||
### 優點
|
||||
|
||||
#### 效能優點
|
||||
- **資料量最小**: 僅包含基本 metadata，儲存效率高
|
||||
- **搜尋速度最快**: 簡單結構提供最佳搜索性能
|
||||
- **實作最簡單**: 易於開發和維護
|
||||
|
||||
#### 多語言優點
|
||||
- **原生多語言支持**: 使用 `nomic-embed-text-v2-moe:latest` 模型，無需語言檢測
|
||||
- **跨語言搜索**: 支持查詢語言與內容語言不同的場景
|
||||
- **統一向量空間**: 所有語言共享相同的 768 維向量空間
|
||||
- **語言無關相似度**: 不同語言描述相同概念的內容會被匹配
|
||||
|
||||
#### 模型優點
|
||||
- **高效推理**: MoE 架構提供快速嵌入生成
|
||||
- **統一維度**: 固定 768 維，與 Qdrant collection 完美匹配
|
||||
- **Ollama 集成**: 通過標準 API 調用，部署簡單
|
||||
|
||||
### 限制
|
||||
|
||||
#### 功能限制
|
||||
- **無法利用物體辨識結果**: 僅基於文本內容，不包含視覺信息
|
||||
- **無法進行父子層級搜尋**: 不支持層級結構的複雜查詢
|
||||
|
||||
#### 模型限制
|
||||
- **固定向量維度**: 僅支持 768 維向量，無法調整
|
||||
- **模型依賴**: 依賴 Ollama 服務運行 `nomic-embed-text-v2-moe:latest` 模型
|
||||
- **多語言精度**: 對於極少數語言可能精度較低
|
||||
|
||||
---
|
||||
|
||||
## 相關文件
|
||||
|
||||
| 文件 | 用途 |
|
||||
|------|------|
|
||||
| CHUNK_RULES_SPEC.md | 規則總覽 |
|
||||
| CHUNK_RULE_2_FRAME_OBJECTS.md | Rule 2 規範 |
|
||||
| CHUNK_RULE_3_COMPOSITE.md | Rule 3 規範 |
|
||||
| CHUNK_SPEC.md | Chunk 基礎規範 |
|
||||
|
||||
---
|
||||
|
||||
**文件結束**
|
||||
@@ -0,0 +1,310 @@
|
||||
---
|
||||
document_type: "reference_doc"
|
||||
service: "MOMENTRY_CORE"
|
||||
title: "Unknown"
|
||||
date: "2026-03-28"
|
||||
version: "V1.0"
|
||||
status: "active"
|
||||
owner: "Warren"
|
||||
created_by: "OpenCode"
|
||||
tags:
|
||||
- "unknown"
|
||||
ai_query_hints:
|
||||
- "查詢 Unknown 的內容"
|
||||
- "Unknown 的主要目的是什麼？"
|
||||
- "如何操作或實施 Unknown？"
|
||||
---
|
||||
|
||||
---
|
||||
title: Chunk Rule 2 - Frame Objects
|
||||
description: 涵蓋 frames，conf > 0.8 的物件加入字串
|
||||
version: 1.0
|
||||
created: 2026-03-28
|
||||
updated: 2026-03-28
|
||||
service: MOMENTRY_CORE
|
||||
topic: chunk_rule
|
||||
document_type: spec
|
||||
rule_id: 2
|
||||
rule_name: Frame Objects
|
||||
collection: momentry_rule2
|
||||
confidence_threshold: 0.8
|
||||
ai_agent_friendly: true
|
||||
---
|
||||
|
||||
# Chunk Rule 2 - Frame Objects
|
||||
|
||||
| 項目 | 內容 |
|
||||
|------|------|
|
||||
| 建立者 | OpenCode |
|
||||
| 建立時間 | 2026-03-28 |
|
||||
| 文件版本 | V1.0 |
|
||||
| Rule ID | 2 |
|
||||
| Rule 名稱 | Frame Objects |
|
||||
| Collection | `momentry_rule2` |
|
||||
| Confidence Threshold | > 0.8 |
|
||||
|
||||
---
|
||||
|
||||
## 版本歷史
|
||||
|
||||
| 版本 | 日期 | 目的 | 操作人 |
|
||||
|------|------|------|--------|
|
||||
| V1.0 | 2026-03-28 | 創建 Rule 2 規範 | OpenCode |
|
||||
|
||||
---
|
||||
|
||||
## 概述
|
||||
|
||||
Rule 2 (Frame Objects) 將 chunk 時間範圍內的 frame objects 進行聚合，僅保留 confidence > 0.8 的物件，轉化為文字描述存入 payload。
|
||||
|
||||
---
|
||||
|
||||
## 設計原則
|
||||
|
||||
### 輸入
|
||||
|
||||
- pre_chunk（來自 ASR/Cut/TimeBased 的原始分段）
|
||||
- frames（chunk 時間範圍內的所有 frames）
|
||||
|
||||
### 處理
|
||||
|
||||
1. 找出 chunk 時間範圍內的所有 frames
|
||||
2. 收集每個 frame 的物件識別結果（YOLO/Face/Pose）
|
||||
3. 過濾 confidence > 0.8 的物件
|
||||
4. 聚合物件名稱和數量
|
||||
|
||||
### 輸出
|
||||
|
||||
- chunk + frame_objects 字串
|
||||
|
||||
---
|
||||
|
||||
## Collection 定義
|
||||
|
||||
### 建立 Collection
|
||||
|
||||
```bash
|
||||
curl -X PUT "http://localhost:6333/collections/momentry_rule2" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: <API_KEY>" \
|
||||
-d '{
|
||||
"vectors": {
|
||||
"size": 768,
|
||||
"distance": "Cosine"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Collection 參數
|
||||
|
||||
| 參數 | 值 |
|
||||
|------|-----|
|
||||
| Name | `momentry_rule2` |
|
||||
| Vector Size | 768 |
|
||||
| Distance | Cosine |
|
||||
| HNSW | m=16, ef_construct=100 |
|
||||
|
||||
---
|
||||
|
||||
## Payload 結構
|
||||
|
||||
### 欄位定義
|
||||
|
||||
| 欄位 | 類型 | 必填 | 說明 |
|
||||
|------|------|------|------|
|
||||
| `uuid` | String | ✅ | 影片 UUID (16 字元) |
|
||||
| `chunk_id` | String | ✅ | Chunk 唯一 ID |
|
||||
| `chunk_type` | String | ✅ | 類型：sentence/cut/time_based |
|
||||
| `chunk_index` | u32 | ✅ | Chunk 索引 (從 0 開始) |
|
||||
| `start_frame` | i64 | ✅ | 開始幀編號 |
|
||||
| `end_frame` | i64 | ✅ | 結束幀編號 |
|
||||
| `fps` | f64 | ✅ | 幀率 |
|
||||
| `original_text` | String | ✅ | 產生 vector 的原始文字 (ASR) |
|
||||
| `frame_objects` | String | ✅ | 涵蓋 frames 的物件描述 (conf > 0.8) |
|
||||
|
||||
### JSON 範例
|
||||
|
||||
```json
|
||||
{
|
||||
"uuid": "1636719dc31f78ac",
|
||||
"chunk_id": "sentence_0001",
|
||||
"chunk_type": "sentence",
|
||||
"chunk_index": 1,
|
||||
"start_frame": 252,
|
||||
"end_frame": 378,
|
||||
"fps": 24.0,
|
||||
"original_text": "Hello world, this is a test message",
|
||||
"frame_objects": "person:3, car:1, dog:2"
|
||||
}
|
||||
```
|
||||
|
||||
### Rust 結構
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct VectorPayloadRule2 {
|
||||
pub uuid: String,
|
||||
pub chunk_id: String,
|
||||
pub chunk_type: String,
|
||||
pub chunk_index: u32,
|
||||
pub start_frame: i64,
|
||||
pub end_frame: i64,
|
||||
pub fps: f64,
|
||||
pub original_text: String,
|
||||
pub frame_objects: String,
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## frame_objects 生成規則
|
||||
|
||||
### 輸入數據（Frame 範例）
|
||||
|
||||
```json
|
||||
// Frame 252
|
||||
{
|
||||
"frame_number": 252,
|
||||
"yolo_objects": [
|
||||
{"class": "person", "confidence": 0.95, "count": 2},
|
||||
{"class": "car", "confidence": 0.85, "count": 1}
|
||||
],
|
||||
"face_count": 0,
|
||||
"pose_count": 0
|
||||
}
|
||||
|
||||
// Frame 300
|
||||
{
|
||||
"frame_number": 300,
|
||||
"yolo_objects": [
|
||||
{"class": "person", "confidence": 0.92, "count": 3},
|
||||
{"class": "dog", "confidence": 0.88, "count": 2}
|
||||
],
|
||||
"face_count": 1,
|
||||
"pose_count": 1
|
||||
}
|
||||
```
|
||||
|
||||
### 處理邏輯
|
||||
|
||||
1. **收集所有物件**：person, car, dog, face, pose
|
||||
2. **過濾 confidence > 0.8**：
|
||||
- person: 0.95 ✅, 0.92 ✅ → 保留
|
||||
- car: 0.85 ✅ → 保留
|
||||
- dog: 0.88 ✅ → 保留
|
||||
- (其他低於 0.8 的過濾掉)
|
||||
3. **聚合數量**：
|
||||
- person: max(2, 3) = 3
|
||||
- car: 1
|
||||
- dog: 2
|
||||
|
||||
### 輸出字串
|
||||
|
||||
```
|
||||
"person:3, car:1, dog:2"
|
||||
```
|
||||
|
||||
### 物件類型前綴
|
||||
|
||||
| 來源 | 前綴 | 範例 |
|
||||
|------|------|------|
|
||||
| YOLO | (class name) | "person:3, car:1" |
|
||||
| Face | "face:" | "face:2" |
|
||||
| Pose | "pose:" | "pose:1" |
|
||||
|
||||
---
|
||||
|
||||
## 時間計算
|
||||
|
||||
### Frame 轉時間
|
||||
|
||||
```
|
||||
start_time = start_frame / fps
|
||||
end_time = end_frame / fps
|
||||
```
|
||||
|
||||
### 範例
|
||||
|
||||
```
|
||||
- fps = 24.0
|
||||
- start_frame = 252
|
||||
- end_frame = 378
|
||||
- start_time = 252 / 24.0 = 10.5 秒
|
||||
- end_time = 378 / 24.0 = 15.75 秒
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 搜尋範例
|
||||
|
||||
### 語義搜尋（包含物件）
|
||||
|
||||
```rust
|
||||
let query_vector = embed_text("找出有人在開車的片段").await?;
|
||||
let results = qdrant.search(
|
||||
"momentry_rule2",
|
||||
&query_vector,
|
||||
10,
|
||||
None
|
||||
).await?;
|
||||
```
|
||||
|
||||
### 依賴 Frame Objects Filter
|
||||
|
||||
```bash
|
||||
curl -X POST "http://localhost:6333/collections/momentry_rule2/points/search" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "api-key: <API_KEY>" \
|
||||
-d '{
|
||||
"vector": [0.123, -0.456, ...],
|
||||
"limit": 10,
|
||||
"with_payload": true,
|
||||
"filter": {
|
||||
"should": [
|
||||
{"key": "frame_objects", "match": {"value": "car"}},
|
||||
{"key": "frame_objects", "match": {"value": "person"}}
|
||||
]
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 使用場景
|
||||
|
||||
| 場景 | 適用 |
|
||||
|------|------|
|
||||
| 需要物體辨識結果輔助搜尋 | ✅ 最佳 |
|
||||
| 根據影片中的物件（人、車、動物）搜尋 | ✅ 最佳 |
|
||||
| 簡單問答系統 + 物件辨識 | ✅ 最佳 |
|
||||
| 需要父子層級關係 | ❌ 請用 Rule 3 |
|
||||
|
||||
---
|
||||
|
||||
## 優點與限制
|
||||
|
||||
### 優點
|
||||
|
||||
- 結合 ASR + 物件辨識結果
|
||||
- 可根據物件進行搜尋
|
||||
- 資料量適中
|
||||
|
||||
### 限制
|
||||
|
||||
- 無法進行父子層級搜尋
|
||||
- frame_objects 是聚合後的字串，無法取得詳細位置
|
||||
|
||||
---
|
||||
|
||||
## 相關文件
|
||||
|
||||
| 文件 | 用途 |
|
||||
|------|------|
|
||||
| CHUNK_RULES_SPEC.md | 規則總覽 |
|
||||
| CHUNK_RULE_1_SIMPLE.md | Rule 1 規範 |
|
||||
| CHUNK_RULE_3_COMPOSITE.md | Rule 3 規範 |
|
||||
| CHUNK_SPEC.md | Chunk 基礎規範 |
|
||||
|
||||
---
|
||||
|
||||
**文件結束**
|
||||
216
docs_v1.0/CHUNKING/RULES/VISUAL_BASED/CHUNK_RULE_2_VISUAL.md
Normal file
216
docs_v1.0/CHUNKING/RULES/VISUAL_BASED/CHUNK_RULE_2_VISUAL.md
Normal file
@@ -0,0 +1,216 @@
|
||||
---
|
||||
document_type: "architecture_design"
|
||||
service: "MOMENTRY_CORE"
|
||||
title: "Momentry Core Chunk Rule 2: 畫面物件級檢索 (Visual Frame Chunk) (v1.0)"
|
||||
date: "2026-04-21"
|
||||
version: "V1.0"
|
||||
status: "active"
|
||||
owner: "Warren"
|
||||
created_by: "OpenCode"
|
||||
tags:
|
||||
- "momentry"
|
||||
- "core"
|
||||
- "rule"
|
||||
- "chunk"
|
||||
ai_query_hints:
|
||||
- "查詢 Momentry Core Chunk Rule 2: 畫面物件級檢索 (Visual Frame Chunk) (v1.0) 的內容"
|
||||
- "Momentry Core Chunk Rule 2: 畫面物件級檢索 (Visual Frame Chunk) (v1.0) 的主要目的是什麼？"
|
||||
- "如何操作或實施 Momentry Core Chunk Rule 2: 畫面物件級檢索 (Visual Frame Chunk) (v1.0)？"
|
||||
---
|
||||
|
||||
# Momentry Core Chunk Rule 2: 畫面物件級檢索 (Visual Frame Chunk) (v1.0)
|
||||
|
||||
| 項目 | 內容 |
|
||||
|------|------|
|
||||
| 建立者 | OpenCode |
|
||||
| 建立時間 | 2026-04-21 |
|
||||
| 文件版本 | V1.0 |
|
||||
|
||||
---
|
||||
|
||||
## 版本歷史
|
||||
|
||||
| 版本 | 日期 | 目的 | 操作人 | 工具/模型 |
|
||||
|------|------|------|--------|-----------|
|
||||
| V1.0 | 2026-04-21 | 定義 Rule 2: 單一幀（或關鍵幀聚合）的數據結構與搜尋邏輯 | OpenCode | OpenCode / Qwen3.6-Plus |
|
||||
|
||||
---
|
||||
|
||||
## 0. 設計目標
|
||||
|
||||
**Rule 2** 的核心概念是**「視覺語義」**。針對影片畫面中出現的具體物件、場景特徵進行精確索引，以支援「畫面內容搜尋」(Visual Search)。
|
||||
|
||||
- **核心原則**: 一個視覺幀 (或短時窗聚合) = 一個 Chunk。
|
||||
- **過濾閾值**: 僅包含 YOLO 信心值 **> 0.8** 的物件，確保索引品質。
|
||||
- **多模態融合**: 結合 YOLO (物件) + Face (人物) + ASRX (說話者)。
|
||||
|
||||
---
|
||||
|
||||
## 1. 數據源與聚合邏輯
|
||||
|
||||
Rule 2 的生成主要依賴視覺處理器產出，並輔助以音訊元數據。
|
||||
|
||||
1. **YOLO (Primary)**: 提供幀級別的物件檢測。
|
||||
- *過濾*: 僅保留 `confidence > 0.8` 的物件。
|
||||
- *格式*: 轉換為描述性字串，如 "a person", "a car", "a cup"。
|
||||
2. **Face (Secondary)**: 提供幀級別的人物 ID。
|
||||
- *策略*: 記錄當前幀所有可見的 `face_id`。
|
||||
3. **ASRX (Audio Overlay)**: 提供當前時間點的說話者。
|
||||
- *策略*: 取時間重疊的 `speaker_id`，若無則為空。
|
||||
|
||||
### 聚合策略 (Time-Windowing)
|
||||
由於影片幀率極高 (如 60fps)，直接為每一幀建立 Chunk 會造成資料庫膨脹。系統採用 **1 秒聚合 (1s Aggregation)** 策略：
|
||||
|
||||
- **Input**: 60 幀 (假設 60fps)。
|
||||
- **Processing**: 合併這 1 秒內所有唯一的 YOLO 物件與 Faces。
|
||||
- **Output**: 1 個 Rule 2 Chunk。
|
||||
- **Time Representation**: 記錄該秒的起始幀 (`start_frame`) 與結束幀 (`end_frame`)。
|
||||
|
||||
---
|
||||
|
||||
## 2. Chunk 結構定義
|
||||
|
||||
### 2.1 資料庫結構 (PostgreSQL)
|
||||
|
||||
```sql
|
||||
CREATE TABLE chunks_rule2 (
|
||||
id UUID PRIMARY KEY,
|
||||
asset_uuid UUID NOT NULL,
|
||||
chunk_type VARCHAR(20) DEFAULT 'visual_frame',
|
||||
|
||||
-- 時間軸 (幀為權威)
|
||||
start_frame INT NOT NULL, -- 聚合區塊起始幀
|
||||
end_frame INT NOT NULL, -- 聚合區塊結束幀
|
||||
start_time_sec DOUBLE PRECISION, -- 參考值
|
||||
end_time_sec DOUBLE PRECISION, -- 參考值
|
||||
fps DOUBLE PRECISION NOT NULL,
|
||||
|
||||
-- 視覺內容 (由 YOLO 產生)
|
||||
content TEXT NOT NULL, -- 描述文本: "car, person, traffic light"
|
||||
frame_objects JSONB, -- 原始物件結構: [{"class": "car", "conf": 0.95}]
|
||||
|
||||
-- 關聯元數據
|
||||
speaker_id VARCHAR(50), -- 當下說話者 (若有)
|
||||
face_ids JSONB, -- 當下出現的人物 ID
|
||||
|
||||
-- 向量與索引
|
||||
embedding vector(768), -- nomic-embed-text-v2-moe
|
||||
search_vector tsvector, -- BM25
|
||||
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
```
|
||||
|
||||
### 2.2 JSON 產出範例
|
||||
|
||||
```json
|
||||
{
|
||||
"chunk_id": "550e...0002",
|
||||
"type": "visual_frame",
|
||||
"content": "car, person, road sign, building",
|
||||
"start_frame": 600,
|
||||
"end_frame": 659,
|
||||
"start_time_sec": 10.00,
|
||||
"end_time_sec": 10.99,
|
||||
"metadata": {
|
||||
"frame_objects": [
|
||||
{ "class": "car", "confidence": 0.98, "box": [10, 10, 50, 50] },
|
||||
{ "class": "person", "confidence": 0.95, "box": [100, 100, 40, 80] }
|
||||
],
|
||||
"faces": ["face_id_01"],
|
||||
"speaker": "SPEAKER_01"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 搜尋能力定義
|
||||
|
||||
Rule 2 專為**視覺語意 (Visual Semantics)** 設計。
|
||||
|
||||
### 3.1 視覺關鍵字搜尋 (Visual Keyword Search)
|
||||
* **場景**: "找出有車子的畫面"、"搜尋開車場景"。
|
||||
* **邏輯**:
|
||||
1. Query: "driving a car"。
|
||||
2. Embedding: 將 "driving a car" 轉為向量。
|
||||
3. Match: 與 `content` ("car, person...") 的向量進行比對。
|
||||
- *注意*: 雖然使用者搜尋是自然語言，但 Rule 2 的底層索引是物件標籤。由於 `nomic-v2-moe` 具有強大的語意對齊能力，"driving a car" 會高度匹配 "car" 標籤。
|
||||
|
||||
### 3.2 高信心值過濾 (Confidence Filtering)
|
||||
* **場景**: "找出 100% 確定有槍的畫面"。
|
||||
* **邏輯**:
|
||||
- 直接查詢 `frame_objects` JSONB 欄位，要求 `confidence > 0.95`。
|
||||
|
||||
### 3.3 跨模態搜尋
|
||||
* **場景**: "找出 Cary Grant 說話且背景有車的畫面"。
|
||||
* **邏輯**:
|
||||
- `face_ids` 包含 "Cary Grant" **AND**
|
||||
- `frame_objects` 包含 "car"。
|
||||
|
||||
---
|
||||
|
||||
## 4. 處理流程 (Processing Pipeline)
|
||||
|
||||
### 4.1 聚合演算法 (Pseudocode)
|
||||
|
||||
```python
|
||||
# 設定: FPS = 30, WINDOW = 30 frames (1 second)
|
||||
|
||||
for i in range(0, total_frames, WINDOW):
|
||||
window_frames = frames[i : i + WINDOW]
|
||||
|
||||
all_objects = []
|
||||
all_faces = set()
|
||||
|
||||
# 1. 遍歷視窗內的幀
|
||||
for frame in window_frames:
|
||||
# YOLO 過濾: 只取信心值 > 0.8
|
||||
valid_objects = [obj for obj in frame.yolo if obj.conf > 0.8]
|
||||
all_objects.extend(valid_objects)
|
||||
|
||||
# Face 收集
|
||||
if frame.faces:
|
||||
all_faces.update([f.id for f in frame.faces])
|
||||
|
||||
# 2. 建立內容摘要 (Content)
|
||||
# 提取唯一類別標籤: "car, person, dog"
|
||||
unique_classes = list(set([obj["class"] for obj in all_objects]))
|
||||
content_desc = ", ".join(unique_classes)
|
||||
|
||||
# 3. 取得該時間段的 Speaker
|
||||
speaker = get_speaker_at_frame(i, asrx_data)
|
||||
|
||||
# 4. 建立 Rule 2 Chunk
|
||||
chunk = {
|
||||
"content": content_desc,
|
||||
"start_frame": i,
|
||||
"end_frame": i + WINDOW - 1,
|
||||
"frame_objects": all_objects, # 保留原始結構供精確過濾
|
||||
"face_ids": list(all_faces),
|
||||
"speaker_id": speaker
|
||||
}
|
||||
|
||||
store_chunk_rule2(chunk)
|
||||
```
|
||||
|
||||
### 4.2 嵌入策略 (Embedding Strategy)
|
||||
|
||||
* **輸入文本**: 僅使用 `content` (物件標籤字串)。
|
||||
* **原因**: 確保向量空間專注於**視覺語意**。若混入 Audio (ASR) 文本，會導致搜尋 "車" 時意外匹配到只提到車但未出現車的畫面。
|
||||
|
||||
---
|
||||
|
||||
## 5. 總結
|
||||
|
||||
Rule 2 提供了**視覺層面**的精確檢索能力，與 Rule 1 (聽覺/語句) 形成互補。
|
||||
|
||||
| 特性 | 實作方式 |
|
||||
|------|----------|
|
||||
| **粒度** | 幀級聚合 (通常為 1 秒區塊) |
|
||||
| **資料過濾** | 僅納入 YOLO Confidence > 0.8 的物件 |
|
||||
| **核心內容** | 物件類別標籤 (Object Tags) |
|
||||
| **人物標記** | 包含 Face ID 與 Speaker ID |
|
||||
| **適用場景** | 尋找特定物件 (槍、車)、場景識別、特定鏡頭回顧 |
|
||||
|
||||
此規範確保了影片畫面中的所有高可信度物件都能被系統「看見」並「記住」。
|
||||
Reference in New Issue
Block a user