momentry_core/docs_v1.0/REFERENCE/DETECTOR_REGISTRY.md

Momentry Core — Detector Registry

Date: 2026-05-13 Version: 1.0 Purpose: 所有模型/演算法檢測器的座標約定、轉換鏈、驗證狀態統整

---

原則

1. 每 detector 一條：獨立記錄輸入/輸出格式、座標原點、單位、轉換公式。
2. 原始座標系標註：不隱藏轉換，任何異於 Top-Left pixel 的輸出必須明列。
3. 轉換鏈可追溯：從 detector 原始輸出到入庫欄位，每一步轉換都記錄。
4. 驗證狀態三級：verified（已測試） / assumed（文檔推斷，未實測） / buggy（已知有誤）。

---

分類總覽

Category	數量	Active	Experimental	Deprecated
face	8	2	4	2
body	3	1	2	0
object	4	1	3	0
text	3	1	2	0
speech	3	2	1	0
scene	2	1	0	1
stamps	2	0	2	0
Total	25	8	14	3

Status	定義
Active	生產 pipeline 中執行，ProcessorType 有註冊，產出被消費
Experimental	獨立腳本或 CLI，不連 pipeline；評估中或備用
Deprecated	評估後棄用；或已被新版取代但未從 codebase 移除

---

Pipeline Status Quick-Reference

#	Detector ID	Short Name	Pipeline Status	Reason
1	DET-CUT-001	PySceneDetect	active	CUT processor
2	DET-SCN-001	Places365	active but rejected ⚠️	M5 eval rejected; never removed from ProcessorType
3	DET-ASR-001	faster-whisper	active	ASR processor
4	DET-SPCH-003	ECAPA-TDNN	active	ASRX speaker embedding
5	DET-OBJ-001	YOLOv8s	active	YOLO processor (v5nu→v8s, 2026-05-13)
6	DET-TEXT-001	swift_ocr	active	OCR processor (primary)
7	DET-FACE-001/002/003	swift_face + FaceNet	active	Face processor
8	DET-BODY-001/002	swift_pose + YOLOv8-pose	active	Pose processor (primary + fallback)
9	DET-FACE-006	AgglomerativeClustering	active	Identity Agent (post-processing)
10	DET-TEXT-005	llama.cpp embed	active	Text embedding (chunk vectors)
11	DET-FACE-005	InsightFace	experimental	Not in production ProcessorType
12	DET-FACE-007	MediaPipe BlazeFace	experimental	MPS fallback, tested but not primary
13	DET-FACE-008	MediaPipe Face Mesh	experimental	Lip processor, not in main pipeline
14	DET-BODY-003	MediaPipe Holistic	experimental	Tested, not in production
15	DET-OBJ-003	OWL-ViT	experimental	Tested for stamps, not in pipeline
16	DET-OBJ-004	Grounding DINO	experimental	Tested for stamps/objects
17	DET-TEXT-002	Florence-2	experimental	Tested for stamps
18	DET-OBJ-002	Gun Detector	experimental	Evaluated, all FP, rejected for pipeline
19	DET-STP-001	OpenCV Stamp	experimental	Used in scan scripts only
20	DET-STP-002	Pose Action Decoder	experimental	Derived from pose, standalone
21	DET-FACE-004	DeepFace ArcFace	deprecated	Replaced by CoreML FaceNet
22	DET-SPCH-002	Apple Speech ASR	deprecated	Replaced by faster-whisper
23	DET-SCN-001	Places365 (scene)	⚠️ deprecated per eval	Still in ProcessorType, needs removal
24	DET-TEXT-003	EmbeddingGemma	experimental	Text embed endpoint, not primary
25	DET-TEXT-004	mxbai CoreML	experimental	Text embed endpoint, not primary

---

Known Misjudgments in Existing Evaluations

#	Evaluation	Issue	Impact	Action
M1	Scene Classification (2026-05-07)	M5 evaluated and REJECTED Places365. But it was never removed from ProcessorType::all(). Still runs on every file.	Wastes ~2min per registration. Produces meaningless scene.json.	Remove from pipeline or re-evaluate
M2	Face Processor benchmark (2026-04-28)	Compared InsightFace vs MediaPipe vs OpenCV vs Contract v1. But the final pipeline uses swift_face + FaceNet, a completely different solution not in the benchmark.	Selection criteria from benchmark don't apply to actual pipeline detector.	Document the actual selection decision for swift_face
M3	Gun Detector (2026-05-07)	Properly rejected: 7/7 FP. Correct decision. Model files still in repo.	No impact (correctly excluded). Clean up model files.	Archive or remove models/gun/
M4	OCR processor	No selection document exists. swift_ocr chosen without comparison against EasyOCR/PaddleOCR.	Unknown if optimal. PaddleOCR fallback may never trigger.	Document selection decision

---

技術分類（有空間座標 vs 無）

Category	數量	有空間座標	僅 Embedding	純時間/文字
face	8	5	3	—
body	3	3	—	—
object	4	4	—	—
text	3	1	2	—
speech	3	—	2	1
scene	2	—	1	1
stamps	2	2	—	—
Total	25	15	8	2

---

Face Detectors

DET-FACE-001 — Face Bbox (Apple Vision)

Field	Value
Framework	Apple Vision
Model	VNDetectFaceRectanglesRequest
Input	CVPixelBuffer (BGRA, via CGImage)
Output	bbox: x, y, width, height
Coordinate	Input: normalized [0-1], origin bottom-left
Transform	x = bb.origin.x * imgW
	y = (1.0 - bb.origin.y - bb.size.height) * imgH
Image size	cgImage.width / cgImage.height
Target	Top-Left pixel integer
File	scripts/swift_processors/swift_face.swift:134-136
Status	✅ verified (2026-05-13, landmark QC + visual check)

---

DET-FACE-002 — Face Landmarks (Apple Vision)

Field	Value
Framework	Apple Vision
Model	VNDetectFaceLandmarksRequest
Input	CVPixelBuffer (BGRA, via CGImage)
Output	landmarks: left_eye (6pt), right_eye (6pt), nose (8pt), outer_lips, inner_lips
Coordinate	Input: VNFaceLandmarks2D.pointsInImage(imageSize:)
	Returned: macOS AppKit convention → bottom-left origin ⚠️
Transform	y_top_left = imgH - $0.y (Y-flip)
Image size	cgImage.width / cgImage.height
Target	Top-Left pixel float → JSON
Pairing	Not by array index. Landmark observations used as primary source (self-consistent bbox + landmarks). Face rect observations deduplicated via IoU > 0.3.
File	scripts/swift_processors/swift_face.swift:155-184
Status	✅ verified (2026-05-13, Y-flip fix, 100% landmark-in-bbox)
Bugs fixed	BUG-001: index-based pairing (landmarkObs[idx] ≠ faceObs[idx])
	BUG-002: macOS bottom-left Y axis (missing Y-flip)

---

DET-FACE-003 — Face Embedding (CoreML FaceNet)

Field	Value
Framework	CoreML (ANE-accelerated)
Model	models/facenet512.mlpackage
Input	Face crop 160×160, RGB, normalized [-1, 1]
Output	512-dim float embedding
Coordinate	N/A (no spatial output). Bbox from DET-FACE-001 used for crop.
File	scripts/face_processor.py, scripts/embed_faces.py, scripts/tmdb_embed_extractor.py
Embedding space	[-1, 1] per dimension, cosine similarity for matching
Status	✅ verified (routinely used for identity matching)

---

DET-FACE-004 — Face Embedding (DeepFace ArcFace)

Field	Value
Framework	DeepFace / TensorFlow
Model	ArcFace (512-dim)
Input	Face crop (from bbox), BGR, no explicit normalization
Output	512-dim float embedding
Coordinate	N/A
File	scripts/face_embedding_extractor.py
Status	🟡 assumed (legacy fallback, not primary pipeline)

---

DET-FACE-005 — Face Recognition (InsightFace)

Field	Value
Framework	InsightFace / ONNX Runtime
Model	buffalo_l (detection + recognition + 5-point landmarks)
Input	Video frame (BGR, numpy array)
Output	bbox: [x1, y1, x2, y2] pixel int
	landmarks: 5-point (left_eye, right_eye, nose, mouth_left, mouth_right)
	embedding: 512-dim float
Coordinate	Bbox: Top-Left pixel (InsightFace native)
	Landmarks: normalized [0-1] to image size
Transform	Bbox: face.bbox.astype(int) — direct
	Landmarks: kps * imgW, kps * imgH — needs manual conversion ⚠️
File	scripts/face_recognition_processor.py:123-153
Status	🟡 assumed (landmark pixel conversion chain not independently verified)

---

DET-FACE-006 — Face Clustering (sklearn)

Field	Value
Framework	sklearn
Model	AgglomerativeClustering
Input	512-dim face embeddings from DET-FACE-003 or DET-FACE-004
Output	cluster labels, centroids (512-dim float)
Coordinate	N/A (no spatial output)
File	scripts/face_clustering_processor.py, scripts/identity_bind.py
Status	✅ verified (428 clusters for Charade, identity_bindings created)

---

DET-FACE-007 — Face Detection (MediaPipe BlazeFace)

Field	Value
Framework	MediaPipe / MPS
Model	blaze_face_short_range.tflite
Input	Frame (numpy array / MPS image)
Output	bbox: [x, y, width, height] pixel
	6 keypoints: eyes, nose tip, mouth center, ear tragions — pixel
Coordinate	Top-Left pixel (MediaPipe native)
Transform	Direct, no conversion needed
File	scripts/face_processor_mps.py
Status	🟡 assumed (MPS fallback, rarely used in pipeline)

---

DET-FACE-008 — Lip Detection (MediaPipe Face Mesh)

Field	Value
Framework	MediaPipe
Model	Face Mesh (468 landmarks)
Input	Face crop or full frame
Output	lip_openness: [0-1] (vertical/mouth_width)
	mouth keypoints: indices 13, 14, 61, 291 from 468 mesh
Coordinate	Landmarks: normalized [0-1], Top-Left origin
Transform	Normalized → pixel: x * imgW, y * imgH
	Lip openness: derived ratio, unitless
File	scripts/lip_processor.py
Status	🟡 assumed

---

Body Pose Detectors

DET-BODY-001 — Body Pose (Apple Vision)

Field	Value
Framework	Apple Vision
Model	VNDetectHumanBodyPoseRequest
Input	CGImage (from frame export or NSImage)
Output	19 keypoints: nose, eyes, ears, neck, root, shoulders, elbows, wrists, hips, knees, ankles
	bbox: [x, y, width, height] derived from keypoint min/max
Coordinate	Input: normalized [0-1], origin bottom-left
Transform (current)	✅ y = h - location.y * h — Y-flip applied
Transform (correct)	y = h - location.y * h
Image size	cgImage.width / cgImage.height
Target	Top-Left pixel float
File	scripts/swift_processors/swift_pose.swift:154-159
Status	✅ verified (2026-05-13, Y-flip fix applied)

---

DET-BODY-002 — Body Pose (YOLOv8 Pose fallback)

Field	Value
Framework	ultralytics / PyTorch
Model	yolov8n-pose.pt
Input	Frame (PIL or numpy)
Output	17 COCO keypoints: nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles
	bbox: [x, y, width, height] derived from keypoints (conf > 0.1)
Coordinate	Top-Left pixel (YOLO native, .xy[0] → numpy float)
Transform	Direct: x, y = float(kps[j][0]), float(kps[j][1])
	Bbox: min(xs), min(ys), max(xs)-min(xs), max(ys)-min(ys)
File	scripts/pose_processor.py:78-97
Status	✅ top-left native

---

DET-BODY-003 — Full Body (MediaPipe Holistic)

Field	Value
Framework	MediaPipe
Model	Holistic (pose + face mesh + hands)
Input	Frame (BGR numpy)
Output	468 face mesh: [[x, y, z], ...] normalized [0-1]
	33 body pose: [[x, y, z, visibility], ...] normalized [0-1]
	21 hand × 2: [[x, y, z], ...] normalized [0-1]
Coordinate	normalized [0-1], Top-Left origin
Transform	x * imgW, y * imgH → pixel (if needed)
	Z: depth relative, not metric
File	scripts/mediapipe_holistic_processor.py
Status	✅ top-left native, normalized→pixel straightforward

---

Object Detectors

DET-OBJ-001 — Object Detection (YOLOv8s)

Field	Value
Framework	ultralytics / CoreML + PyTorch fallback
Model	yolov8s.mlpackage (primary, CoreML ANE), yolov8s.pt (fallback)
mAP (COCO)	44.9 (was 34.3 with YOLOv5nu, +31%)
Input	Frame (PIL or numpy)
Output	bbox: [x1, y1, x2, y2] — float pixel
	class_name, class_id (80 COCO classes)
	confidence: [0-1]
Coordinate	Top-Left pixel (YOLO .xyxy[0] → float)
Transform	Rust: x = detection.x1 as i32, y = detection.y1 as i32 — int truncation
	width = x2 - x1, height = y2 - y1
Image size	YOLO auto-handles via ultralytics inference
File	scripts/yolo_processor.py:272-285, src/core/processor/yolo.rs:83-117
Status	✅ verified (2026-05-13, replaced YOLOv5nu, +19% detections, scene indicators +162~+473%)
Replaced	YOLOv5nu (mAP 34.3, removed 2026-05-13)

---

DET-OBJ-002 — Weapon Detection (YOLOv8n Fine-tuned)

Field	Value
Framework	ultralytics / PyTorch
Model	models/gun/gun_detector/weights/best.pt
Input	Frame (numpy array)
Output	bbox: [x1, y1, x2, y2] pixel
	class: {0: grenade, 1: knife, 2: pistol, 3: rifle}
Coordinate	Top-Left pixel (YOLO native)
File	scripts/gun_detector_scan.py
Status	✅ top-left native

---

DET-OBJ-003 — Open-Vocabulary Detection (OWL-ViT)

Field	Value
Framework	HuggingFace Transformers
Model	google/owlvit-base-patch32
Input	PIL Image + text queries
Output	bbox, scores, labels
Coordinate	post_process_object_detection returns boxes in [x1, y1, x2, y2] format
	scaled to target_sizes parameter
Transform	target_sizes = torch.Tensor([image_pil.size[::-1]]) — PIL (w,h) → (h,w)
	box.int().tolist() or box.tolist() → Python list
Format risk	HuggingFace processor version may return [cx, cy, w, h] not [x1,y1,x2,y2]
File	scripts/test_owl_vit_stamps.py:69-80, scripts/magnifying_glass_owl.py:65-77
Status	🟡 assumed (bbox format not independently verified with visual check)
Verify	Render bbox overlay on a known target image, confirm x1 < x2, y1 < y2

---

DET-OBJ-004 — Open-Vocabulary Detection (Grounding DINO)

Field	Value
Framework	HuggingFace Transformers
Model	IDEA-Research/grounding-dino-base
Input	PIL Image + text prompts
Output	boxes, labels, scores
Coordinate	processor rescales to target_sizes, returns pixel boxes
Transform	target_sizes=[img.size[::-1]] — PIL (w,h) → (h,w)
	[round(v, 1) for v in dets["boxes"][i].tolist()]
Format risk	[::-1] order depends on processor expectations. If processor expects (w,h), axes swapped.
File	scripts/gdino_frame_api.py:176-180
Status	🟡 assumed (rescale direction not independently verified)
Verify	Single-frame output: check bbox x range ≤ imgW, y range ≤ imgH

---

Text / OCR Detectors

DET-TEXT-001 — OCR (Apple Vision)

Field	Value
Framework	Apple Vision
Model	VNRecognizeTextRequest (accurate/fast)
Input	CVPixelBuffer (via CGImage)
Output	text: string, bbox: [x, y, w, h], confidence: [0-1]
Coordinate	Input: VNRecognizedTextObservation.boundingBox — normalized [0-1], origin bottom-left
Transform	✅ y = (1.0 - bb.origin.y - bb.size.height) * cgH — Y-flip applied
Image size	Main loop: cgImage.width / cgImage.height ✅
	recognizeText() helper: CVPixelBufferGetWidth/Height ✅
File	scripts/swift_processors/swift_ocr.swift:125-133, :181-182
Status	✅ verified (2026-05-13, Y-flip + image size fix applied)

---

DET-TEXT-002 — Open-Vocabulary (Florence-2)

Field	Value
Framework	HuggingFace Transformers
Model	microsoft/Florence-2-base
Input	PIL Image + task prompt
Output	bbox: [x1, y1, x2, y2] pixel
	label, text (depending on task)
Coordinate	processor post_process_generation rescales to image_size, returns pixel
Transform	x1, y1, x2, y2 = map(int, bbox) — direct
	image_size=(image_pil.width, image_pil.height) — (w, h) order ✅
File	scripts/florence2_scan_stamps.py:67-79, scripts/test_florence2_direct.py
Status	✅ top-left native (HuggingFace post_process output)

---

DET-TEXT-003 — Text Embedding (EmbeddingGemma)

Field	Value
Framework	HuggingFace / PyTorch MPS
Model	google/embeddinggemma-300m
Input	Text string
Output	Embedding vector (L2 normalized, dimension model-dependent)
Coordinate	N/A
File	scripts/embeddinggemma_server.py
Status	✅ verified (embedding API server)

---

Text Embedding (Non-Detector)

DET-TEXT-004 — Text Embedding (mxbai CoreML)

Field	Value
Framework	CoreML (ANE-accelerated)
Model	mxbai-embed-large-v1.mlpackage
Input	Text tokenized
Output	Embedding vector
Coordinate	N/A
File	scripts/coreml_embed_server.py
Status	🟡 assumed

---

DET-TEXT-005 — Text Embedding (Ollama / llama.cpp)

Field	Value
Framework	llama.cpp / Ollama API
Model	llama.cpp embedding endpoint (port 11436)
Input	Text (optionally prefixed search_document:)
Output	768-dim float embedding
Coordinate	N/A
File	src/core/embedding/comic_embed.rs
Status	✅ verified (embedding pipeline)

---

Speech / Audio Detectors

DET-SPCH-001 — ASR (faster-whisper)

Field	Value
Framework	faster-whisper / CTranslate2
Model	faster-whisper/small (int8 CPU)
Input	Audio extracted from video
Output	[{start, end, text}, ...] — temporal segments (seconds)
Coordinate	Temporal only (seconds), no spatial
File	scripts/asr_processor.py
Status	✅ verified (ASR pipeline)

---

DET-SPCH-002 — ASR (Apple Speech)

Field	Value
Framework	Apple Speech (ANE)
Model	SFSpeechRecognizer
Input	Audio file
Output	[{start, end, text, confidence}, ...] — temporal segments
Coordinate	Temporal only (seconds), no spatial
File	scripts/swift_processors/asr_swift.swift
Status	🟡 assumed (Apple Speech quality lower than faster-whisper)

---

DET-SPCH-003 — Speaker Embedding (ECAPA-TDNN)

Field	Value
Framework	SpeechBrain / PyTorch
Model	speechbrain/spkrec-ecapa-voxceleb
Input	Audio segments per speaker
Output	192-dim float embedding
Coordinate	N/A (vector space, cosine similarity)
File	scripts/asrx_processor_custom.py, scripts/voice_embedding_extractor.py
Status	✅ verified (voice embeddings exported to SQLite + Qdrant)

---

Scene Detectors

DET-SCN-001 — Scene Classification (Places365)

Field	Value
Framework	CoreML (ANE) + PyTorch MPS fallback
Model	resnet18_places365.mlpackage
Input	Frame resized to 224×224
Output	[{scene_type, confidence, top_5}, ...] — temporal segments
Coordinate	Temporal only, no spatial
File	scripts/scene_classifier.py
Status	✅ verified

---

DET-SCN-002 — Scene Cut Detection (PySceneDetect)

Field	Value
Framework	PySceneDetect
Model	ContentDetector (threshold-based frame difference)
Input	Video frames
Output	[{scene_number, start_frame, end_frame, start_time, end_time}]
Coordinate	Temporal (frames + seconds), no spatial
File	scripts/cut_processor.py
Status	✅ verified

---

Stamp / Specific Target Detectors

DET-STP-001 — Stamp Detection (OpenCV Color)

Field	Value
Framework	OpenCV
Model	HSV color masking + contour analysis (rule-based, no ML)
Input	Frame (BGR numpy)
Output	bbox: [x, y, w, h] pixel
Coordinate	Top-Left pixel (cv2.boundingRect() native)
Transform	Direct, no conversion
File	scripts/scan_full_video_stamps.py, scripts/find_blue_stamp_opencv.py
Status	✅ top-left native

---

DET-STP-002 — Pose Action Decoder (Coordinate-derived)

Field	Value
Framework	Rule-based from keypoints
Model	N/A (derived from DET-BODY-001/002/003 keypoints)
Input	Pose keypoints (pixel)
Output	Action labels: turn_left, turn_right, look_up, look_down, shake_head, nod_head, blink, smile, etc.
Coordinate	Derived angles/ratios, no raw spatial output
File	scripts/utils/pose_action_decoder.py, scripts/utils/integrated_body_action_decoder.py
Status	🟡 assumed (actions derived from pose keypoints; dependent on upstream keypoint correctness)
Warning	Affected by DET-BODY-001 Y-flip bug — all action labels wrong when using Vision pose

---

Known Bugs Summary

Bug ID	Detector	Issue	Impact	Fixed
BUG-001	DET-FACE-001/002	Index-based landmark↔face pairing	Wrong landmarks assigned to wrong faces	✅ 2026-05-13
BUG-002	DET-FACE-002	macOS bottom-left → missing Y-flip	Landmarks 731px offset from bbox	✅ 2026-05-13
BUG-003	DET-BODY-001	Missing Y-flip on keypoints	All 19 joint Y coordinates inverted	✅ 2026-05-13
BUG-004	DET-BODY-001	Derived bbox Y inverted	Bbox doesn't cover actual person	✅ 2026-05-13
BUG-005	DET-TEXT-001	Missing Y-flip on bbox	Text bbox Y inverted	✅ 2026-05-13
BUG-006	DET-TEXT-001	Hardcoded 640×360 in recognizeText()	Wrong bbox scale for non-640×360 images	✅ 2026-05-13

---

Coordinate Convention Quick Reference

Apple Vision (all detectors)

Item	Convention
boundingBox origin	Bottom-Left
boundingBox units	normalized [0-1]
pointsInImage Y axis	Bottom-Left (macOS AppKit)
Required Y-flip formula	bbox: y = (1 - y_norm - h_norm) * imgH
	points: y = imgH - raw_y

Non-Vision Detectors

Framework	Origin	Units
YOLO (ultralytics)	Top-Left	pixel float
MediaPipe	Top-Left	normalized [0-1]
InsightFace bbox	Top-Left	pixel int
InsightFace landmarks	Top-Left	normalized [0-1]
HuggingFace (post_process)	Top-Left	pixel (after rescale)
OpenCV	Top-Left	pixel int

---

納管規則

1. 新增 detector：必須在此 Registry 註冊，含座標系、轉換公式、檔案位置。
2. 座標變更：任何轉換公式修改，必須更新此文件並標註變更日期。
3. 驗證要求：每個有空間座標的 detector 必須通過至少一次 visual check（bbox/keypoints 疊加原圖）。
4. 跨 detector 比對：同一 frame 的不同 detector 輸出 bbox，IoU 應合理（非零且非 1.0）。
5. Vision detector 鐵律：任何使用 Apple Vision Framework 的 detector，必須確認 Y-flip 已實作。

---

維護

• Owner: M5
• 更新頻率: 每次新增 processor 或修改座標轉換時
• 參照: SPATIAL_COORDINATE_REGISTRY.md（上層座標系統）