diff --git a/docs_v1.0/DESIGN/LaunchDaemon_Config_M5Max128.md b/docs_v1.0/DESIGN/LaunchDaemon_Config_M5Max128.md new file mode 100644 index 0000000..2cafac2 --- /dev/null +++ b/docs_v1.0/DESIGN/LaunchDaemon_Config_M5Max128.md @@ -0,0 +1,421 @@ +--- +title: LaunchDaemon Architecture (M5Max128 Reference) +version: 1.0 +date: 2026-05-27 +author: M5Max128 +status: reference +--- + +# LaunchDaemon Architecture Reference + +> **Scope**: M5Max128 local configuration (resource-managed binaries) +> **Note**: M5Max48 uses build-from-source approach via start_momentry.sh. Both approaches are valid and independent. + +## Overview + +| Machine | Approach | Status | +|---------|----------|--------| +| M5Max128 | LaunchDaemon + resource binaries | Reference document | +| M5Max48 | start_momentry.sh + build from source | Main branch | + +## Architecture Principles + +``` +/Library/LaunchDaemons/ (system-level, boot before login) + ├── com.momentry.postgresql.plist (P1, no dependency) + ├── com.momentry.redis.plist (P1, no dependency) + ├── com.momentry.qdrant.plist (P2, no dependency) + ├── com.momentry.mongodb.plist (P2, no dependency) + └── com.momentry.gitea.plist (P3, depends on PostgreSQL) + +Experimental services: + └── com.momentry.startup.plist (LLM, Embedding, Playground, etc.) +``` + +## Key Design Points + +### 1. Binary Location + +All binaries are resource-managed under `/Users/accusys/momentry_resources/bin/`: + +| Service | Binary Path | +|---------|-------------| +| PostgreSQL | `/Users/accusys/pgsql/18.3/bin/postgres` | +| Redis | `/Users/accusys/momentry_resources/bin/redis-server` | +| Qdrant | `/Users/accusys/momentry_resources/bin/qdrant` | +| MongoDB | `/Users/accusys/momentry_resources/bin/mongod` | +| Gitea | `/Users/accusys/momentry_resources/bin/gitea` | + +### 2. Root Boot → User Execution + +LaunchDaemons run at boot (root), but use `UserName` key to switch to user: + +```xml +UserName +accusys +``` + +### 3. Unified Log Path + +All logs go to `/Users/accusys/momentry/logs/`: + +```xml +StandardOutPath +/Users/accusys/momentry/logs/.log + +StandardErrorPath +/Users/accusys/momentry/logs/.error.log +``` + +## Plist Templates + +### PostgreSQL + +```xml + + + + + Label + com.momentry.postgresql + + UserName + accusys + + WorkingDirectory + /Users/accusys/momentry/var/postgresql + + ProgramArguments + + /Users/accusys/pgsql/18.3/bin/postgres + -D + /Users/accusys/momentry/var/postgresql + + + RunAtLoad + + + KeepAlive + + + StandardOutPath + /Users/accusys/momentry/logs/postgresql.log + + StandardErrorPath + /Users/accusys/momentry/logs/postgresql.error.log + + +``` + +### Redis (ACL Authentication) + +```xml + + + + + Label + com.momentry.redis + + UserName + accusys + + WorkingDirectory + /Users/accusys/momentry/var/redis + + ProgramArguments + + /Users/accusys/momentry_resources/bin/redis-server + --port + 6379 + --bind + 0.0.0.0 + --aclfile + /Users/accusys/momentry/etc/redis/users.acl + --dir + /Users/accusys/momentry/var/redis + --logfile + /Users/accusys/momentry/logs/redis.log + + + RunAtLoad + + + KeepAlive + + + StandardOutPath + /Users/accusys/momentry/logs/redis.log + + StandardErrorPath + /Users/accusys/momentry/logs/redis.error.log + + +``` + +### Redis ACL File + +Location: `/Users/accusys/momentry/etc/redis/users.acl` + +``` +user default on sanitize-payload ~* &* +@all >accusys +user accusys on sanitize-payload ~* &* +@all >accusys +``` + +**Redis 8.x Authentication**: +```bash +# Old (deprecated): redis-cli -a accusys ping +# New (recommended): redis-cli --user default --pass accusys ping +``` + +### Qdrant + +```xml + + + + + Label + com.momentry.qdrant + + UserName + accusys + + WorkingDirectory + /Users/accusys/momentry/var/qdrant/ + + ProgramArguments + + /Users/accusys/momentry_resources/bin/qdrant + + + EnvironmentVariables + + QDRANT__STORAGE__STORAGE_PATH + /Users/accusys/momentry/var/qdrant/ + QDRANT__SERVICE__HOST + 0.0.0.0 + QDRANT__SERVICE__HTTP_PORT + 6333 + HOME + /Users/accusys + + + RunAtLoad + + + KeepAlive + + + StandardOutPath + /Users/accusys/momentry/logs/qdrant.log + + StandardErrorPath + /Users/accusys/momentry/logs/qdrant.error.log + + +``` + +### MongoDB + +```xml + + + + + Label + com.momentry.mongodb + + UserName + accusys + + ProgramArguments + + /Users/accusys/momentry_resources/bin/mongod + --dbpath + /Users/accusys/momentry/var/mongodb + --logpath + /Users/accusys/momentry/logs/mongodb.log + --port + 27017 + --bind_ip + 0.0.0.0 + + + RunAtLoad + + + KeepAlive + + + StandardOutPath + /Users/accusys/momentry/logs/mongodb.log + + StandardErrorPath + /Users/accusys/momentry/logs/mongodb.error.log + + WorkingDirectory + /Users/accusys/momentry/var/mongodb + + +``` + +### Gitea (with Wrapper Script) + +```xml + + + + + Label + com.momentry.gitea + + UserName + accusys + + WorkingDirectory + /Users/accusys/momentry/var/gitea + + ProgramArguments + + /Users/accusys/momentry_core/scripts/start_gitea.sh + + + EnvironmentVariables + + HOME + /Users/accusys + GITEA_WORK_DIR + /Users/accusys/momentry/var/gitea + + + RunAtLoad + + + KeepAlive + + + StandardOutPath + /Users/accusys/momentry/logs/gitea.log + + StandardErrorPath + /Users/accusys/momentry/logs/gitea.error.log + + +``` + +## Wrapper Script: start_gitea.sh + +Gitea depends on PostgreSQL. Wrapper script ensures PostgreSQL is ready: + +```bash +#!/bin/bash + +PG_BIN="/Users/accusys/pgsql/18.3/bin" +GITEA_BIN="/Users/accusys/momentry_resources/bin/gitea" +GITEA_CONFIG="/Users/accusys/momentry/etc/gitea/app.ini" + +MAX_WAIT=60 +WAITED=0 + +# Wait for PostgreSQL +while ! "$PG_BIN/pg_isready" -q 2>/dev/null; do + if [ $WAITED -ge $MAX_WAIT ]; then + echo "ERROR: PostgreSQL not ready after $MAX_WAIT seconds" + exit 1 + fi + sleep 2 + WAITED=$((WAITED + 2)) +done + +# Start Gitea +"$GITEA_BIN" web --config "$GITEA_CONFIG" +``` + +## Install Script: install_launchdaemons.sh + +```bash +#!/bin/bash + +PLIST_DIR="/Users/accusys/momentry_core/momentry_runtime/plist" +DAEMON_DIR="/Library/LaunchDaemons" +LOG_DIR="/Users/accusys/momentry/logs" + +mkdir -p "$LOG_DIR" + +DAEMONS=( + "com.momentry.postgresql" + "com.momentry.redis" + "com.momentry.qdrant" + "com.momentry.mongodb" + "com.momentry.gitea" +) + +for daemon in "${DAEMONS[@]}"; do + plist_name="${daemon}.plist" + src="${PLIST_DIR}/${plist_name}" + dest="${DAEMON_DIR}/${plist_name}" + + if launchctl list "$daemon" >/dev/null 2>&1; then + sudo launchctl unload -w "$dest" 2>/dev/null + fi + + sudo cp "$src" "$dest" + sudo chown root:wheel "$dest" + sudo chmod 644 "$dest" + sudo launchctl load -w "$dest" +done +``` + +## Comparison: M5Max128 vs M5Max48 + +| Aspect | M5Max128 | M5Max48 | +|--------|----------|---------| +| **Approach** | LaunchDaemon (system-level) | start_momentry.sh (user script) | +| **Binaries** | Resource-managed (`momentry_resources/bin/`) | Build from source (`services/*/target/`) | +| **PostgreSQL data** | `/Users/accusys/momentry/var/postgresql` | `/Users/accusys/pgsql/data` | +| **Redis auth** | ACL file (`users.acl`) | `--requirepass` (deprecated) | +| **LLM path** | Resource binary | `/Users/accusys/llama/bin/` | +| **Gitea** | Independent LaunchDaemon | Not in startup script | +| **MongoDB** | Independent LaunchDaemon | Not in startup script | + +## Installation Steps (M5Max128) + +```bash +# 1. Ensure directories exist +mkdir -p /Users/accusys/momentry/logs +mkdir -p /Users/accusys/momentry/var/{postgresql,redis,qdrant,mongodb,gitea} + +# 2. Install LaunchDaemons (requires sudo) +sudo /Users/accusys/momentry_core/scripts/install_launchdaemons.sh + +# 3. Verify services +/Users/accusys/pgsql/18.3/bin/pg_isready +/Users/accusys/momentry_resources/bin/redis-cli --user default --pass accusys ping +curl http://localhost:6333/healthz +curl http://localhost:3000/ + +# 4. Reboot test +sudo reboot + +# 5. Post-reboot verification +launchctl list | grep com.momentry +``` + +## Notes + +1. **Independence**: M5Max128's LaunchDaemons do not conflict with M5Max48's startup script. Each machine has its own approach. + +2. **Resource Management**: M5Max128 uses pre-built binaries from `momentry_resources/bin/`, avoiding build dependencies. + +3. **Redis ACL**: Redis 8.x uses ACL authentication, not `--requirepass`. This is the modern approach. + +4. **Gitea Wrapper**: Essential because Gitea depends on PostgreSQL. The wrapper ensures PostgreSQL is ready before starting Gitea. + +--- + +## Version History + +| Version | Date | Author | Changes | +|---------|------|--------|---------| +| 1.0 | 2026-05-27 | M5Max128 | Initial reference document | \ No newline at end of file diff --git a/docs_v1.0/DESIGN/Processor_Refactoring_Assessment.md b/docs_v1.0/DESIGN/Processor_Refactoring_Assessment.md new file mode 100644 index 0000000..a6c6972 --- /dev/null +++ b/docs_v1.0/DESIGN/Processor_Refactoring_Assessment.md @@ -0,0 +1,352 @@ +--- +title: Processor Refactoring Assessment (M5Max128 Research) +version: 1.0 +date: 2026-05-27 +author: M5Max128 +status: reference +--- + +# Processor Refactoring Assessment + +> **Scope**: M5Max128 research documentation for M5Max48 implementation reference +> **Workspace**: ~/workspace/ (22 modules) + +## Executive Summary + +22 processor modules evaluated for Rust/Swift/Python refactoring feasibility. + +### Priority Matrix + +| Phase | Language | Modules | Effort | Benefit | +|-------|----------|---------|--------|---------| +| 1 | Swift | OCR, Pose, Face | Low | Remove Python wrappers | +| 2 | Rust | TKG, Resume, Redis | Low | Remove infrastructure deps | +| 3 | Rust | Cut | Medium | Pure CPU logic | +| 4 | Swift | YOLO | Medium | ANE acceleration | +| 5 | Python | Others (keep) | - | ML/LLM dependencies | + +--- + +## Phase 1: Swift Modules (Immediate Gain) + +### workspace_ocr + +| Metric | Value | +|--------|-------| +| Swift Suitability | 10/10 | +| Current State | Thin Python wrapper around swift_ocr | +| Refactoring | Delete Python wrapper, Rust calls swift_ocr directly | +| LOC Change | Python: -122, Rust: ~50 | +| Risk | Low | +| Effort | 1 day | + +**Current Architecture**: +``` +Rust (ocr.rs) → PythonExecutor → ocr_processor.py → subprocess → swift_ocr +``` + +**Target Architecture**: +``` +Rust (ocr.rs) → subprocess → swift_ocr +``` + +### workspace_pose + +| Metric | Value | +|--------|-------| +| Swift Suitability | 10/10 | +| Current State | Thin Python wrapper around swift_pose | +| Refactoring | Delete Python wrapper, Rust calls swift_pose directly | +| LOC Change | Python: -150, Rust: ~50 | +| Risk | Low | +| Effort | 1 day | + +**Current Architecture**: +``` +Rust (pose.rs) → PythonExecutor → pose_processor.py → subprocess → swift_pose +``` + +**Target Architecture**: +``` +Rust (pose.rs) → subprocess → swift_pose +``` + +### workspace_face + +| Metric | Value | +|--------|-------| +| Swift Suitability | 9/10 | +| Current State | Swift detect + Python embedding (FaceNet CoreML) | +| Refactoring | Merge detection + embedding into single Swift binary | +| LOC Change | Python: -337, Swift: +100 (embedding) | +| Risk | Medium | +| Effort | 2-3 days | + +**Current Architecture**: +``` +Stage 1: Python → swift_face (Vision detect) → bbox + landmarks +Stage 2: Python → OpenCV crop → CoreML FaceNet → 512D embedding +``` + +**Target Architecture**: +``` +Swift: Vision detect → crop → VNCoreMLModel (FaceNet) → embedding → face.json +``` + +### workspace_face_recognition + +| Metric | Value | +|--------|-------| +| Status | **Superseded** | +| Recommendation | Do not refactor. Archive/remove. | +| Note | Replaced by face_processor.py (Apple Vision + CoreML) | + +--- + +## Phase 2: Rust Modules (Infrastructure) + +### workspace_tkg + +| Metric | Value | +|--------|-------| +| Rust Suitability | **10/10** | +| Current State | Python psycopg2 + SQL queries | +| Dependencies | PostgreSQL, JSON I/O (no ML) | +| Refactoring | Pure Rust with sqlx/tokio-postgres | +| LOC Change | Python: -469, Rust: ~350 | +| Risk | Low | +| Effort | 1-2 days | + +**Graph Structure**: +``` +NODES: + (face_trace) - one per trace_id + (object) - one per YOLO class + (speaker) - one per speaker_id + +EDGES: + (face) -[:CO_OCCURS_WITH]-> (object) same frame + (face) -[:SPEAKS_AS]-> (speaker) temporal overlap + (face) -[:CO_OCCURS_WITH]-> (face) same frame +``` + +### workspace_resume_framework + +| Metric | Value | +|--------|-------| +| Rust Suitability | **10/10** | +| Current State | Python file I/O + signal handling | +| Dependencies | File I/O, timers (no ML) | +| Refactoring | Pure Rust struct with auto-save | +| LOC Change | Python: -484, Rust: ~150 | +| Risk | Low | +| Effort | 1 day | + +**Rust Design**: +```rust +struct ResumeFramework { + path: PathBuf, + save_interval: Duration, + last_save: Instant, + position: Option, +} + +impl ResumeFramework { + fn load_checkpoint(&mut self) -> Result> + fn save_checkpoint(&self, position: u64) -> Result<()> + fn auto_save_tick(&mut self, position: u64) -> Result + fn finalize(&mut self, total: u64) -> Result<()> +} +``` + +### workspace_redis_publisher + +| Metric | Value | +|--------|-------| +| Rust Suitability | **10/10** | +| Current State | Python redis-py pub/sub | +| Dependencies | Redis TCP (no ML) | +| Refactoring | Pure Rust with redis-rs | +| LOC Change | Python: -195, Rust: ~100 | +| Risk | Low | +| Effort | 1 day | + +**Rust Design**: +```rust +use redis::AsyncCommands; + +struct ProgressPublisher { + client: redis::Client, + channel: String, +} + +impl ProgressPublisher { + async fn info(&self, processor: &str, msg: &str) -> Result<()> + async fn progress(&self, processor: &str, current: u32, total: u32, msg: &str) -> Result<()> + async fn complete(&self, processor: &str, msg: &str) -> Result<()> + async fn error(&self, processor: &str, msg: &str) -> Result<()> +} +``` + +--- + +## Phase 3: Rust CPU Logic + +### workspace_cut + +| Metric | Value | +|--------|-------| +| Rust Suitability | 8/10 | +| Current State | Python PySceneDetect | +| Dependencies | Pure CPU (histogram diff) | +| Refactoring | Port ContentDetector algorithm to Rust | +| LOC Change | Python: -106, Rust: ~300 | +| Risk | Medium | +| Effort | 2-3 days | +| Challenge | HSV histogram + adaptive threshold | + +**Algorithm to Port**: +- Frame-to-frame HSV/Luma histogram difference +- Rolling average threshold +- min_scene_len enforcement + +--- + +## Phase 4: Swift ANE Acceleration + +### workspace_yolo + +| Metric | Value | +|--------|-------| +| Swift Suitability | 8/10 | +| Current State | Python ultralytics (YOLOv8) | +| Dependencies | CoreML model conversion needed | +| Refactoring | Create swift_yolo with VNCoreMLModel | +| LOC Change | Python: -496, Swift: ~300 | +| Risk | Medium | +| Effort | 2-3 days | +| Challenge | CoreML model conversion, async handling | + +**Swift Approach**: +1. Convert YOLOv8 → CoreML: `yolo export model=yolov8s.pt format=coreml` +2. Create swift_yolo.swift with VNCoreMLModel +3. AVAssetReader for frame extraction +4. ANE-accelerated inference + +--- + +## Phase 5: Python Keep (ML/LLM Dependencies) + +### Modules to Keep in Python + +| Module | Reason | +|--------|--------| +| asr | whisper/faster-whisper (no Rust/Swift equivalent) | +| asrx | speaker diarization (pyannote) | +| audio_taxonomy | librosa/tensorflow | +| lip | MediaPipe lip tracking | +| caption | LLM generation | +| scene | ML scene classification | +| story | LLM generation | +| story_pipeline | LLM pipeline | +| tmdb_agent | API agent | +| identity_agent | LLM agent | +| voice_embedding | ML embedding | +| mediapipe_holistic | MediaPipe (no Rust/Swift binding) | +| visual_chunk | Visual processing | + +--- + +## Implementation Roadmap + +### Week 1: Swift Wrapper Removal + +1. OCR: Modify `ocr.rs` to call swift_ocr directly +2. Pose: Modify `pose.rs` to call swift_pose directly +3. Test both with sample videos + +### Week 2: Rust Infrastructure + +4. redis_publisher: Create `src/core/redis_publisher.rs` +5. resume_framework: Create `src/core/resume.rs` +6. TKG: Create `src/core/processor/tkg.rs` + +### Week 3: Swift Enhancement + +7. Face: Extend swift_face.swift with CoreML embedding +8. Test face embedding pipeline + +### Week 4: Rust Algorithm Port + +9. Cut: Port ContentDetector to Rust +10. Test scene detection + +### Week 5: Swift ANE + +11. YOLO: Convert yolov8s → CoreML +12. Create swift_yolo.swift +13. Test object detection + +--- + +## Total Effort Estimate + +| Phase | LOC (Rust/Swift) | Effort | +|-------|------------------|--------| +| 1 | ~100 | 1-2 days | +| 2 | ~600 | 3-4 days | +| 3 | ~100 | 2-3 days | +| 4 | ~300 | 2-3 days | +| 5 | ~300 | 2-3 days | +| **Total** | ~1400 | **10-15 days** | + +--- + +## Dependency Removal Summary + +| Dependency | Removed By | +|------------|------------| +| Python runtime | All Swift/Rust refactors | +| redis-py | redis_publisher (Rust) | +| psycopg2 | TKG (Rust) | +| PySceneDetect | Cut (Rust) | +| ultralytics (YOLO) | swift_yolo | +| OpenCV (face crop) | Face Swift embedding | +| InsightFace | Already superseded | + +--- + +## Appendix: Module Summary Table + +| Module | Language | Suitability | Status | Action | +|--------|----------|-------------|--------|--------| +| ocr | Swift | 10/10 | Active | Delete wrapper | +| pose | Swift | 10/10 | Active | Delete wrapper | +| face | Swift | 9/10 | Active | Extend Swift | +| face_recognition | - | - | Superseded | Archive | +| yolo | Swift | 8/10 | Active | Create Swift | +| cut | Rust | 8/10 | Active | Port algorithm | +| tkg | Rust | 10/10 | Active | Pure Rust | +| resume_framework | Rust | 10/10 | Active | Pure Rust | +| redis_publisher | Rust | 10/10 | Active | Pure Rust | +| asr | Python | 2/10 | Keep | ML dependency | +| asrx | Python | 2/10 | Keep | ML dependency | +| audio_taxonomy | Python | 2/10 | Keep | ML dependency | +| lip | Python | 2/10 | Keep | ML dependency | +| caption | Python | 2/10 | Keep | LLM | +| scene | Python | 2/10 | Keep | ML | +| story | Python | 2/10 | Keep | LLM | +| story_pipeline | Python | 2/10 | Keep | LLM | +| tmdb_agent | Python | 4/10 | Keep | API | +| identity_agent | Python | 4/10 | Keep | LLM | +| voice_embedding | Python | 2/10 | Keep | ML | +| mediapipe_holistic | Python | 2/10 | Keep | ML | +| visual_chunk | Python | 3/10 | Keep | Visual | + +--- + +## Version History + +| Version | Date | Author | Changes | +|---------|------|--------|---------| +| 1.0 | 2026-05-27 | M5Max128 | Initial assessment from workspace research | \ No newline at end of file diff --git a/docs_v1.0/DESIGN/Thumbnail_JPEG_Validation_Impl.md b/docs_v1.0/DESIGN/Thumbnail_JPEG_Validation_Impl.md new file mode 100644 index 0000000..9257bb2 --- /dev/null +++ b/docs_v1.0/DESIGN/Thumbnail_JPEG_Validation_Impl.md @@ -0,0 +1,279 @@ +--- +title: Thumbnail JPEG Validation Implementation +version: 1.0.0 +date: 2026-05-27 +author: M5Max128 +status: ready_for_implementation +--- + +# Thumbnail JPEG Validation Implementation + +## Overview + +Add JPEG quality validation to all ffmpeg image extraction endpoints to prevent: +- Empty images (0 bytes) +- Corrupted JPEG (missing header/footer) +- Incomplete JPEG (truncated output) + +## Files to Create/Modify + +### 1. Create: `src/core/thumbnail/validator.rs` + +```rust +use anyhow::{bail, Result}; + +pub const JPEG_MIN_SIZE: usize = 100; +pub const JPEG_SOI_MARKER: [u8; 3] = [0xFF, 0xD8, 0xFF]; +pub const JPEG_EOI_MARKER: [u8; 2] = [0xFF, 0xD9]; + +pub fn validate_jpeg(data: &[u8]) -> Result<()> { + if data.len() < JPEG_MIN_SIZE { + bail!("JPEG too small: {} bytes (minimum {})", data.len(), JPEG_MIN_SIZE); + } + + if data[0..3] != JPEG_SOI_MARKER { + bail!("Invalid JPEG header: expected {:02X?}, got {:02X?}", JPEG_SOI_MARKER, &data[0..3]); + } + + if data[data.len() - 2..] != JPEG_EOI_MARKER { + bail!("Incomplete JPEG: missing EOI marker, got {:02X?}", &data[data.len() - 2..]); + } + + Ok(()) +} + +pub fn is_valid_jpeg(data: &[u8]) -> bool { + validate_jpeg(data).is_ok() +} + +pub fn jpeg_size_ok(data: &[u8]) -> bool { + data.len() >= JPEG_MIN_SIZE +} + +pub fn jpeg_header_ok(data: &[u8]) -> bool { + data.len() >= 3 && data[0..3] == JPEG_SOI_MARKER +} + +pub fn jpeg_footer_ok(data: &[u8]) -> bool { + data.len() >= 2 && data[data.len() - 2..] == JPEG_EOI_MARKER +} +``` + +### 2. Modify: `src/core/thumbnail/mod.rs` + +Add module declaration at line 1: + +```rust +pub mod validator; + +use anyhow::{Context, Result}; +// ... rest of file +``` + +### 3. Modify: `src/api/media_api.rs` + +Location: `face_thumbnail()` function, after ffmpeg output check (around line 754) + +Add validation: + +```rust +if !output.status.success() { + return Err(StatusCode::INTERNAL_SERVER_ERROR); +} + +// ADD THIS LINE: +crate::core::thumbnail::validator::validate_jpeg(&output.stdout) + .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?; + +Ok(Response::builder() + // ... rest of response +``` + +### 4. Modify: `src/api/trace_agent_api.rs` + +Location: `get_trace_thumbnail()` function, after reading bytes (around line 544) + +Add validation: + +```rust +let bytes = tokio::fs::read(&tmp).await.map_err(|e| { + (StatusCode::INTERNAL_SERVER_ERROR, Json(serde_json::json!({"error": e.to_string()}))) +})?; + +let _ = tokio::fs::remove_file(&tmp).await; + +// ADD THIS LINE: +crate::core::thumbnail::validator::validate_jpeg(&bytes) + .map_err(|e| { + (StatusCode::INTERNAL_SERVER_ERROR, Json(serde_json::json!({"error": e.to_string()}))) + })?; + +Ok(Response::builder() + // ... rest of response +``` + +### 5. Modify: `src/core/frame_cache.rs` + +Location: `FrameManager::extract()`, when iterating extracted frames (around line 73) + +Replace the frame collection logic: + +```rust +for entry in &entries { + let fname = entry.file_name(); + let fname_str = fname.to_string_lossy(); + if let Some(num_str) = fname_str + .strip_prefix("frame_") + .and_then(|s| s.strip_suffix(".jpg")) + { + if let Ok(frame_num) = num_str.parse::() { + let frame_path = entry.path(); + // ADD VALIDATION: + if let Ok(data) = std::fs::read(&frame_path) { + if crate::core::thumbnail::validator::is_valid_jpeg(&data) { + let timestamp = frame_num as f64 / fps; + frames.push(CachedFrame { + path: frame_path, + frame_number: frame_num, + timestamp_secs: timestamp, + }); + } else { + info!("[FrameCache] Skipping invalid JPEG: {:?}", frame_path); + } + } + } + } +} +``` + +## Python Scripts (Optional Enhancement) + +### 6. Create: `scripts/utils/jpeg_validator.py` + +```python +#!/usr/bin/env python3 +"""JPEG validation utilities for ffmpeg-extracted frames.""" + +JPEG_MIN_SIZE = 100 +JPEG_SOI_MARKER = bytes([0xFF, 0xD8, 0xFF]) +JPEG_EOI_MARKER = bytes([0xFF, 0xD9]) + + +def validate_jpeg(data: bytes) -> bool: + """Validate JPEG by checking header, footer, and minimum size.""" + if len(data) < JPEG_MIN_SIZE: + return False + if data[:3] != JPEG_SOI_MARKER: + return False + if data[-2:] != JPEG_EOI_MARKER: + return False + return True + + +def validate_jpeg_file(path: str) -> bool: + """Validate JPEG file on disk.""" + try: + with open(path, "rb") as f: + data = f.read() + return validate_jpeg(data) + except Exception: + return False + + +def filter_valid_jpegs(paths: list[str]) -> list[str]: + """Filter list of paths to only valid JPEGs.""" + return [p for p in paths if validate_jpeg_file(p)] +``` + +### 7. Modify: `scripts/thumbnail_extractor.py` + +Location: After extracting each thumbnail (around line 65) + +Add validation: + +```python +if result.returncode == 0 and os.path.exists(output_file): + # ADD VALIDATION: + if validate_jpeg_file(output_file): + extracted.append(output_file) + print(f" Extracted: {output_file} at {ts:.1f}s", file=sys.stderr) + else: + print(f" Invalid JPEG at {ts:.1f}s", file=sys.stderr) + os.remove(output_file) # Clean up invalid file +else: + print(f" Failed to extract frame at {ts:.1f}s", file=sys.stderr) +``` + +### 8. Modify: `scripts/caption_processor.py` + +Location: `extract_frames()` function, after ffmpeg extraction (around line 70) + +Add validation: + +```python +try: + subprocess.run(cmd, capture_output=True, check=False) + if os.path.exists(output_file): + # ADD VALIDATION: + if validate_jpeg_file(output_file): + frames.append({"index": i, "timestamp": timestamp, "path": output_file}) + else: + os.remove(output_file) # Clean up invalid file +except Exception: + pass +``` + +### Python Scripts Affected + +| Script | Function | Line | Priority | +|--------|----------|------|----------| +| `thumbnail_extractor.py` | `extract_thumbnails()` | 65 | High (user-facing) | +| `caption_processor.py` | `extract_frames()` | 70 | Medium | +| `caption_processor_contract_v1.py` | `extract_frames()` | 310 | Medium | +| `ocr_processor_contract_v1.py` | `extract_frames()` | 367 | Medium | +| `qa/executor.py` | `extract_frames()` | 93 | Low (QA only) | +| `face_cross_validate.py` | `extract_frames()` | 16 | Low (testing) | +| `face_mediapipe_test.py` | `extract_frames()` | 25 | Low (testing) | +| `analyze_video_faces.py` | `extract_video_frames()` | 61 | Low (analysis) | + +## Validation Logic + +| Check | Condition | Error if failed | +|-------|-----------|-----------------| +| Minimum size | `len() >= 100` | "JPEG too small" | +| SOI marker | `[0..3] == [0xFF,0xD8,0xFF]` | "Invalid JPEG header" | +| EOI marker | `[-2..] == [0xFF,0xD9]` | "Incomplete JPEG" | + +## Testing + +After implementation, run: + +```bash +source ~/.cargo/env +export MOMENTRY_PYTHON_PATH="/Users/accusys/momentry_core/venv/bin/python" +cargo clippy --lib +cargo test --lib +``` + +Expected: 220 passed, 0 failed + +## Commit Message + +``` +feat: add JPEG validation to thumbnail endpoints + +- Create validator module with JPEG header/footer/size checks +- Add validation to face_thumbnail endpoint +- Add validation to get_trace_thumbnail endpoint +- Filter invalid JPEGs in FrameManager::extract +- (Optional) Add Python jpeg_validator utility for script validation + +Prevents serving corrupted/incomplete JPEG images to frontend. +``` + +## Version History + +| Version | Date | Author | Changes | +|---------|------|--------|---------| +| 1.0.0 | 2026-05-27 | M5Max128 | Implementation plan ready | +| 1.1.0 | 2026-05-27 | M5Max128 | Added Python scripts section | \ No newline at end of file diff --git a/docs_v1.0/DESIGN/Thumbnail_QA_Analysis.md b/docs_v1.0/DESIGN/Thumbnail_QA_Analysis.md new file mode 100644 index 0000000..015bb75 --- /dev/null +++ b/docs_v1.0/DESIGN/Thumbnail_QA_Analysis.md @@ -0,0 +1,340 @@ +--- +title: Thumbnail Endpoint Quality Assurance Analysis +version: 1.0.0 +date: 2026-05-27 +author: M5Max128 +status: research_complete +--- + +# Thumbnail Endpoint Quality Assurance Analysis + +## Scope + +| Item | Status | +|------|--------| +| Research | Complete | +| Implementation | Pending (M5Max48) | +| Affected Endpoints | 2 | + +## Overview + +Thumbnail endpoints currently lack quality validation, resulting in potential anomalies: +- **Empty images** - ffmpeg produces 0 bytes output +- **Black frames** - extracted frame is all black +- **Corrupted JPEG** - incomplete ffmpeg output + +## Affected Endpoints + +| Endpoint | File | Line | +|----------|------|------| +| `/api/v1/file/:file_uuid/thumbnail` | `src/api/media_api.rs` | 700-764 | +| `/api/v1/file/:file_uuid/trace/:trace_id/thumbnail` | `src/api/trace_agent_api.rs` | 514-556 | + +--- + +## Anomaly Classification + +### Type 1: Empty Image (No Frame) + +**Symptom**: Returns 0 bytes or very small JPEG + +**Root Causes**: +1. `frame_number > total_frames` - requested frame exceeds video length +2. Video file missing or corrupted +3. Codec does not support frame-level seek +4. ffmpeg `-vf select` filter finds no matching frame + +**Code Locations**: +- `media_api.rs:710-716` - `query_auto_representative_frame()` may return invalid frame +- `media_api.rs:720-728` - `file_path` query may return non-existent file +- `media_api.rs:754-756` - only checks `output.status.success()`, not output content + +### Type 2: Black Frame + +**Symptom**: Returns valid JPEG but all black or very dark + +**Root Causes**: +1. `crop` parameters exceed video dimensions (`x+w > width` or `y+h > height`) +2. Extracted frame is from fade-in/fade-out transition +3. Video has black opening/closing credits +4. Low-light scene + +**Code Locations**: +- `media_api.rs:731-735` - crop validation missing +- `trace_agent_api.rs:530` - crop may exceed dimensions + +### Type 3: Corrupted JPEG + +**Symptom**: Returns incomplete JPEG (browser shows broken image) + +**Root Causes**: +1. ffmpeg stdout pipe interrupted before completion +2. ffmpeg process killed mid-output +3. JPEG encoder failure +4. Incomplete write to stdout buffer + +**Code Locations**: +- `media_api.rs:751` - pipe output may be truncated +- `media_api.rs:758-763` - no JPEG validation before serving + +--- + +## Current Quality Mechanisms + +### Endpoint 1: `face_thumbnail` + +| Mechanism | Status | Location | +|-----------|--------|----------| +| Representative frame selection | Present | `tkg::query_auto_representative_frame()` | +| ffmpeg success check | Present | `output.status.success()` | +| JPEG validation | Missing | - | +| Size validation | Missing | - | +| Black frame detection | Missing | - | +| Retry mechanism | Missing | - | + +### Endpoint 2: `get_trace_thumbnail` + +| Mechanism | Status | Location | +|-----------|--------|----------| +| Blur detection (candidate selection) | Present | `select_rep_face()` lines 463-480 | +| Confidence filter (>0.7) | Present | `select_rep_face()` line 429 | +| QC metadata filter | Present | `select_rep_face()` line 430 | +| ffmpeg success check | Present | `status.status.success()` | +| JPEG validation | Missing | - | +| Black frame detection (extraction) | Missing | - | +| Retry mechanism | Missing | - | + +**Note**: `select_rep_face()` has sophisticated quality control for SELECTING the representative face, but the actual EXTRACTION step lacks validation. + +--- + +## Root Cause Analysis + +### A. Input Data Problems + +| Problem | Impact | Condition | +|---------|--------|-----------| +| `frame_number > total_frames` | Empty image | TKG returns wrong frame, user passes invalid value | +| `crop exceeds dimensions` | Black frame / error | face bbox incorrect, video resolution changed | +| Video file missing | 500 error | File deleted/moved | +| Codec不支持seek | Empty/corrupted | Some codecs only support sequential read | + +### B. ffmpeg Execution Problems + +| Problem | Impact | Cause | +|---------|--------|-------| +| `select` no output | Empty JPEG | frame超出範圍 → ffmpeg skips all frames | +| Pipe interrupted | Corrupted JPEG | stdout buffer full, ffmpeg terminated early | +| `-ss` imprecise | Wrong frame | input seeking approximate, error ±5 frames | +| crop failure | Black frame / 500 | `x+w > width` or `y+h > height` | + +### C. Quality Control Gaps + +| Gap | Impact | Current | +|-----|--------|---------| +| No JPEG validation | Corrupted image served | Only checks exit code | +| No size check | 0 bytes returned | No output length check | +| No black detection | Black frame served | blurdetect only in candidate selection | +| No retry | Single failure = error | No retry mechanism | + +--- + +## Concrete Failure Cases + +### Case 1: Frame Exceeds Range + +``` +Video: total_frames=1000 (DB record) +Actual: video has only 950 frames (file truncated) +Request: frame=980 +ffmpeg: select=eq(n\,980) → no match +Output: 0 bytes JPEG +Frontend: blank image +``` + +### Case 2: Crop Exceeds Dimensions + +``` +Video: 1920x1080 +face_bbox: x=1850, y=1050, w=100, h=100 +ffmpeg: crop=100:100:1850:1050 +Result: x+100=1950 > 1920 → ffmpeg error or black border +``` + +### Case 3: Seek Imprecise + +``` +Video: 25fps +Request: frame=1000 (40 seconds) +ffmpeg -ss 40.0 -i video +Actual: seeks to frame 995~1005 range +Result: extracts different face than select_rep_face chose +``` + +### Case 4: Pipe Interrupted + +``` +ffmpeg -i large_video -vf select=eq(n\,50000) -f image2pipe - +Video large, select needs scan to frame 50000 +Pipe buffer full → ffmpeg may be killed or terminate early +Output: incomplete JPEG (missing FFD9 footer) +``` + +--- + +## Recommended Fixes + +### Phase P0: Critical (Must Implement) + +| Fix | Description | LOC | Location | +|-----|-------------|-----|----------| +| **Frame validation** | `frame <= total_frames` | ~20 | `media_api.rs:707-718` | +| **Crop validation** | `x+w <= width, y+h <= height` | ~15 | `media_api.rs:731-735` | +| **JPEG header check** | `data[0..3] == [0xFF,0xD8,0xFF]` | ~10 | Helper function | +| **JPEG footer check** | `data[-2..] == [0xFF,0xD9]` | ~10 | Helper function | +| **Minimum size check** | `data.len() > 100` | ~5 | Helper function | + +### Phase P1: Important (Should Implement) + +| Fix | Description | LOC | Location | +|-----|-------------|-----|----------| +| **Black frame detection** | ffmpeg `-vf blackdetect` filter | ~30 | After extraction | +| **Output seeking** | Move `-ss` after `-i` for precision | ~5 | `trace_agent_api.rs:527` | + +### Phase P2: Enhancement (Nice to Have) + +| Fix | Description | LOC | Location | +|-----|-------------|-----|----------| +| **Retry mechanism** | Max 3 attempts, offset +30 frames each | ~50 | Both endpoints | +| **Fallback frame** | Extract middle frame if all fail | ~30 | Both endpoints | + +--- + +## Implementation Plan + +### Step 1: Create Validation Module + +Create `src/core/thumbnail/validator.rs`: + +```rust +pub fn validate_jpeg(data: &[u8]) -> Result<()> { + // P0-1: Minimum size + if data.len() < 100 { + bail!("JPEG too small: {} bytes", data.len()); + } + + // P0-2: JPEG header (SOI marker) + if data[0..3] != [0xFF, 0xD8, 0xFF] { + bail!("Invalid JPEG header"); + } + + // P0-3: JPEG footer (EOI marker) + if data[data.len()-2..] != [0xFF, 0xD9] { + bail!("Incomplete JPEG"); + } + + Ok(()) +} +``` + +### Step 2: Add Frame/Crop Validation + +In `media_api.rs`: + +```rust +// P0-4: Validate frame number +let total_frames: i64 = sqlx::query_scalar(...) + .bind(&file_uuid) + .fetch_one(pool) + .await?; + +if frame > total_frames { + return Err(StatusCode::BAD_REQUEST); +} + +// P0-5: Validate crop dimensions +if let (Some(x), Some(y), Some(w), Some(h)) = (q.x, q.y, q.w, q.h) { + let (width, height): (i32, i32) = sqlx::query_as(...) + .bind(&file_uuid) + .fetch_one(pool) + .await?; + + if x + w > width || y + h > height { + return Err(StatusCode::BAD_REQUEST); + } +} +``` + +### Step 3: Integrate Validation + +In both endpoints, after ffmpeg extraction: + +```rust +// Apply validation +validate_jpeg(&output.stdout) + .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?; +``` + +--- + +## Testing Strategy + +### Test Cases + +| Test | Input | Expected | +|------|-------|----------| +| Valid frame | `frame=500` (valid) | JPEG returned | +| Frame exceeds | `frame=999999` | 400 BAD_REQUEST | +| Valid crop | `x=100,y=100,w=200,h=200` | JPEG returned | +| Crop exceeds | `x=1800,y=1000,w=200,h=200` | 400 BAD_REQUEST | +| Empty video | corrupted video file | 500 INTERNAL_ERROR | +| Black frame | fade-out frame | Retry or fallback | + +--- + +## Files to Modify + +| File | Changes | +|------|---------| +| `src/core/thumbnail/mod.rs` | Add validator module | +| `src/core/thumbnail/validator.rs` | New file (validation helpers) | +| `src/api/media_api.rs` | Add validation in `face_thumbnail()` | +| `src/api/trace_agent_api.rs` | Add validation in `get_trace_thumbnail()` | + +--- + +## Estimated Effort + +| Phase | LOC | Time | +|-------|-----|------| +| P0 (Critical) | ~60 | 1-2 days | +| P1 (Important) | ~35 | 1 day | +| P2 (Enhancement) | ~80 | 2-3 days | +| **Total** | ~175 | 4-6 days | + +--- + +## Version History + +| Version | Date | Author | Changes | +|---------|------|--------|---------| +| 1.0.0 | 2026-05-27 | M5Max128 | Initial analysis complete | + +--- + +## Next Steps for M5Max48 + +1. Read this document +2. Implement P0 fixes first +3. Test with edge cases +4. Add P1/P2 as needed +5. Update `AGENTS.md` if adding new validation commands + +--- + +## References + +- `docs_v1.0/DESIGN/Processor_Refactoring_Assessment.md` - Processor refactoring priorities +- `src/api/media_api.rs:700-764` - face_thumbnail implementation +- `src/api/trace_agent_api.rs:394-556` - select_rep_face and get_trace_thumbnail +- `ffmpeg -vf blackdetect` documentation \ No newline at end of file diff --git a/docs_v1.0/OPERATIONS/VERIFICATION_RUSTDESK_2026-05-22.md b/docs_v1.0/OPERATIONS/VERIFICATION_RUSTDESK_2026-05-22.md new file mode 100644 index 0000000..aa1cdd3 --- /dev/null +++ b/docs_v1.0/OPERATIONS/VERIFICATION_RUSTDESK_2026-05-22.md @@ -0,0 +1,26 @@ +# RustDesk Verification Report + +**Date**: 2026-05-22 +**Status**: Verified + +## Source + +| Item | Value | +|------|-------| +| Version | 1.4.6 | +| Source | GitHub release | +| Source repo | `admin/rustdesk` (Gitea) | +| Binary | `~/Applications/RustDesk.app` | +| Install | DMG → copy to ~/Applications | + +## Verification + +| Check | Result | +|-------|--------| +| DMG downloaded | ✅ 24MB, aarch64 | +| .app bundle | ✅ 59MB | +| Architecture | Apple Silicon (arm64) | + +## Linked Documents + +- `docs_v1.0/OPERATIONS/Services_Inventory.md` diff --git a/release/system/v1.0/services/sources_manifest.json b/release/system/v1.0/services/sources_manifest.json index 8279f14..43ef08b 100644 --- a/release/system/v1.0/services/sources_manifest.json +++ b/release/system/v1.0/services/sources_manifest.json @@ -263,6 +263,14 @@ "file_count": 1279, "gitea_repo": "http://192.168.110.200:3000/admin/yt.git", "verification_doc": "docs_v1.0/OPERATIONS/Services_Inventory.md" + }, + { + "name": "rustdesk-1.4.6-aarch64.dmg", + "type": "file", + "size_mb": 24, + "description": "RustDesk remote desktop", + "gitea_repo": "http://192.168.110.200:3000/admin/rustdesk.git", + "verification_doc": "docs_v1.0/OPERATIONS/VERIFICATION_RUSTDESK_2026-05-22.md" } ] } \ No newline at end of file diff --git a/release/system/v1.0/services/src/rustdesk-1.4.6-aarch64.dmg b/release/system/v1.0/services/src/rustdesk-1.4.6-aarch64.dmg new file mode 100644 index 0000000..3540b49 Binary files /dev/null and b/release/system/v1.0/services/src/rustdesk-1.4.6-aarch64.dmg differ