feat: Phase 1 handover - schema migration, correction mechanism, API fixes

Schema changes: dev.chunks->dev.chunk, remove old_chunk_id/chunk_index Correction: asr-1.json format, generate/apply scripts API: 37/37 endpoints fixed and tested Docs: HANDOVER_V2.0.md for M4
2026-05-11 07:03:22 +08:00
parent ef894a44ad
commit 39ba5ddf76
147 changed files with 19843 additions and 3053 deletions
--- a/docs/M5_SETUP_LOG.md
+++ b/docs/M5_SETUP_LOG.md
@@ -0,0 +1,259 @@
+# M5 Dev Environment Setup Log
+
+**Machine**: M5 MacBook Pro (MacOS 26.4.1, Apple M5 Max, 48GB)
+**User**: accusys (admin group, sudo with password)
+**Date**: 2026-05-06
+**Setup by**: OpenCode
+
+---
+
+## 1. Source Code
+
+| Item | Detail |
+|------|--------|
+| Repo | `https://gitea.momentry.ddns.net/warren/momentry_core.git` |
+| Branch | `main` |
+| Commit | `bac6c2d` (feat: identity clustering V3.0) |
+| Sync method | rsync from M4 (192.168.110.210) |
+| Path | `~/momentry_core_0.1/` |
+
+---
+
+## 2. Installed Services
+
+### 2.1 PostgreSQL 18.3
+
+| Field | Value |
+|-------|-------|
+| **Source** | [https://ftp.postgresql.org/pub/source/v18.3/postgresql-18.3.tar.gz](https://ftp.postgresql.org/pub/source/v18.3/postgresql-18.3.tar.gz) |
+| **GitHub** | [https://github.com/postgresql/postgresql](https://github.com/postgresql/postgresql) |
+| **Build method** | Manual `./configure && make && make install` |
+| **Prefix** | `~/pgsql/18.3/` |
+| **Data dir** | `~/pgsql/data/` |
+| **Port** | 5432 |
+| **Version** | PostgreSQL 18.3 |
+| **SHA256** | `ab04939aafdb9e8487c2f13dda91e6a4a7f4c83368f5bedd23ee4ad1fda64afb` |
+| **Start command** | `pg_ctl -D ~/pgsql/data -l ~/pgsql/pg.log start` |
+| **Configure flags** | `--prefix=$HOME/pgsql/18.3 --with-uuid=e2fs --with-icu --with-openssl` |
+| **Build date** | 2026-05-06 |
+| **Notes** | `--with-uuid=e2fs` used (requires Homebrew `e2fsprogs`). macOS built-in UUID not detected by configure. |
+
+### 2.2 pgvector 0.8.2
+
+| Field | Value |
+|-------|-------|
+| **Source** | [https://github.com/pgvector/pgvector](https://github.com/pgvector/pgvector) |
+| **Version** | v0.8.2 |
+| **Build method** | `git clone && make && make install` |
+| **SHA256** | `65dec31ec078d60ee9d8e1dac59be8a41edf8c79bf380cd0093691b0afd257a8` |
+| **Build date** | 2026-05-06 |
+| **Notes** | Built against PostgreSQL 18.3 source installation |
+
+### 2.3 Redis 7.4.3
+
+| Field | Value |
+|-------|-------|
+| **Source** | [https://github.com/redis/redis/archive/refs/tags/7.4.3.tar.gz](https://github.com/redis/redis/archive/refs/tags/7.4.3.tar.gz) |
+| **GitHub** | [https://github.com/redis/redis](https://github.com/redis/redis) |
+| **Version** | 7.4.3 |
+| **Build method** | `make -j$(sysctl -n hw.ncpu)` |
+| **Binary path** | `~/redis/bin/redis-server` |
+| **Port** | 6379 |
+| **SHA256** | `87b6a9ea145c56c1ace724acbb9906b7be4abddd44041545adf44ce9f4d0a615` |
+| **Start command** | `redis-server --daemonize yes --port 6379` |
+| **Build date** | 2026-05-06 |
+
+### 2.4 Qdrant 1.17.1
+
+| Field | Value |
+|-------|-------|
+| **Source** | [https://github.com/qdrant/qdrant.git](https://github.com/qdrant/qdrant.git) |
+| **Version** | v1.17.1 |
+| **Build method** | `cargo build --release --bin qdrant` |
+| **Binary path** | `~/momentry_core_0.1/services/qdrant/target/release/qdrant` |
+| **Storage dir** | `~/qdrant_storage` |
+| **Port** | 6333 (HTTP), 6334 (gRPC) |
+| **SHA256** | `8f8aa63840a0f948b43f9b95f784ace69595892de5dc581bb66bd62fd86d6c66` |
+| **Build date** | 2026-05-06 |
+| **Config** | `~/qdrant_config.yaml` |
+| **Start command** | `qdrant --config-path ~/qdrant_config.yaml &` |
+| **Build deps** | protoc (Homebrew protobuf), cmake |
+
+### 2.5 MongoDB 8.2.7
+
+| Field | Value |
+|-------|-------|
+| **Source** | Homebrew `mongodb/brew/mongodb-community` |
+| **Version** | 8.2.7 |
+| **Port** | 27017 |
+| **Start command** | `brew services start mongodb/brew/mongodb-community` |
+| **Install date** | 2026-05-06 |
+
+### 2.6 MariaDB 12.2.2
+
+| Field | Value |
+|-------|-------|
+| **Source** | Homebrew `mariadb` |
+| **Version** | 12.2.2-MariaDB |
+| **Port** | 3306 |
+| **Start command** | `brew services start mariadb` |
+| **Install date** | 2026-05-06 |
+
+### 2.7 PHP 8.5.5
+
+| Field | Value |
+|-------|-------|
+| **Source** | Homebrew `php` |
+| **Version** | 8.5.5 |
+| **WordPress extensions** | mysqli, pdo_mysql, gd, xml, mbstring, curl, zip, json, intl, bcmath, gmp, openssl |
+| **Start command** | `brew services start php` |
+| **Install date** | 2026-05-06 |
+
+### 2.8 FFmpeg / FFprobe 8.1.1
+
+| Field | Value |
+|-------|-------|
+| **Source** | Homebrew `ffmpeg` |
+| **Version** | 8.1.1 |
+| **SHA256** | `00d01197255300c02122c783dd0126a9e7f47d6c6a19faafae2e6610efd071d3` |
+| **Install date** | 2026-05-06 |
+
+### 2.9 SFTPGo 2.7.1
+
+| Field | Value |
+|-------|-------|
+| **Source** | [https://github.com/drakkan/sftpgo.git](https://github.com/drakkan/sftpgo.git) |
+| **Version** | v2.7.1 |
+| **Build method** | `git clone && go build -o sftpgo_bin ./` |
+| **Binary path** | `~/momentry_core_0.1/services/sftpgo_bin` |
+| **SHA256** | `550b6653f8f2cd7c58620e128e85be571a6702c79cf374824ad9b420ca039db1` |
+| **Build date** | 2026-05-06 |
+| **Patch** | Upgraded `go-m1cpu` from v0.2.0 → v0.2.1 to fix SIGTRAP crash on macOS 26.4.1 |
+| **Notes** | Pre-built binary from GitHub releases crashed with `go-m1cpu` cgo compatibility issue. Source build with patched dependency resolved. |
+
+### 2.10 OpenCode 1.14.39
+
+| Field | Value |
+|-------|-------|
+| **Source** | [https://opencode.ai/install](https://opencode.ai/install) |
+| **Version** | 1.14.39 |
+| **Binary path** | `~/.opencode/bin/opencode` |
+| **SHA256** | `def4a786c257bd6a965e46a2b069802496681b9eea20261d7d1b55629af3d1da` |
+| **Install date** | 2026-05-06 |
+
+### 2.11 Python 3.11 + Packages
+
+| Field | Value |
+|-------|-------|
+| **Source** | Homebrew `python@3.11` |
+| **Version** | 3.11.15 |
+| **Path** | `/opt/homebrew/bin/python3.11` |
+| **Key packages** | coremltools, opencv-python, numpy, psycopg2, torch, transformers, whisperx, etc. |
+| **Requirements** | `~/momentry_core_0.1/requirements.txt` |
+| **Install date** | 2026-05-06 |
+| **FaceNet model** | `models/facenet512.mlpackage` (512D CoreML, loads OK) |
+
+### 2.12 Build Tools
+
+| Tool | Version | Source |
+|------|---------|--------|
+| Rust | 1.95.0 | rustup (pre-installed) |
+| Go | 1.26.2 | Homebrew `go` |
+| cmake | 4.3.2 | Homebrew `cmake` |
+| pkg-config | - | Homebrew `pkg-config` |
+
+---
+
+## 3. Momentry Configuration
+
+### 3.1 Environment Files
+
+| File | Purpose |
+|------|---------|
+| `.env` | Production config (port 3002) |
+| `.env.development` | Development config (port 3003) |
+
+Key settings:
+- `DATABASE_URL=postgres://accusys@localhost:5432/momentry`
+- `REDIS_URL=redis://:accusys@localhost:6379`
+- `DATABASE_SCHEMA=dev`
+- `MOMENTRY_SERVER_PORT=3003` (dev) / `3002` (prod)
+- `MOMENTRY_API_KEY=muser_test_apikey`
+- `MOMENTRY_PYTHON_PATH=/opt/homebrew/bin/python3.11`
+- `MOMENTRY_SCRIPTS_DIR=/Users/accusys/momentry_core_0.1/scripts`
+
+### 3.2 Database Tables Created
+
+| Table | Created by |
+|-------|-----------|
+| `dev.videos` | Manual SQL |
+| `dev.chunks` | Manual SQL |
+| `dev.monitor_jobs` | Manual SQL |
+| `dev.processor_results` | Manual SQL |
+| `dev.talents` | Manual SQL |
+| `dev.identity_bindings` | Manual SQL |
+| `dev.api_keys` | Manual SQL |
+
+### 3.3 API Key
+
+- Key: `muser_test_apikey`
+- Hash (SHA256): `3f2fa16e44ff74267786fdf979b9c33dac0cad515282e4937a0776756a61e821`
+- Status: active
+
+---
+
+## 4. Running Services (Verified)
+
+| Service | Port | Status |
+|---------|------|--------|
+| PostgreSQL | 5432 | ✅ |
+| Redis | 6379 | ✅ |
+| Qdrant | 6333 | ✅ |
+| MongoDB | 27017 | ✅ |
+| MariaDB | 3306 | ✅ |
+| Momentry Playground | 3003 | ✅ |
+| Gemma4 LLM | 8081 | ✅ (pre-installed) |
+
+---
+
+## 5. PATH Configuration
+
+`.zshrc`:
+```zsh
+export PATH="/opt/homebrew/bin:/opt/homebrew/opt/postgresql@18/bin:$HOME/.opencode/bin:$PATH"
+```
+
+Also available:
+- `$HOME/pgsql/18.3/bin` — source-built PostgreSQL tools
+- `$HOME/redis/bin` — source-built Redis
+- `$HOME/.cargo/bin` — Rust/Cargo tools
+
+---
+
+## 6. M5 End-to-End Test Results (Charade Full Movie)
+
+Run date: 2026-05-06 20:38-20:57
+
+| Stage | Time | Result |
+|-------|------|--------|
+| **Swift_face** (Vision ANE detection) | 867s (14.5 min) | 3999 frames (interval=30) |
+| **CoreML FaceNet** (512D embedding) | 271s (4.5 min) | 6186 face embeddings |
+| **Face tracker** (scene-cut aware) | ~30s | 1538 traces |
+| **DB store** | ~5s | 6186 detections in `dev.face_detections` |
+| **Total** | ~19 min | 1 long video (412k frames, 2.2GB) |
+
+**Scene-cut effect**: 1538 traces (vs 379 without scene-cut reset in M4 data). Scene boundaries correctly split traces.
+
+**Models used**:
+- Face detection: Apple Vision (ANE) via `swift_face`
+- Face embedding: CoreML FaceNet 512D via `facenet512.mlpackage`
+- Text embedding: `mxbai-embed-large` (1024D) via Ollama
+
+---
+
+## 7. Known Issues
+
+1. **Momentry API status `degraded`**: Expected on fresh setup. Some cache/processing dependencies not fully initialized.
+2. **SFTPGo startup requires config**: Binary built from source, needs config file for production use.
+3. **Migration scripts not all run**: Base tables created manually. Some migration files (017+) reference tables/columns that need verification.
+4. **OpenCode config**: `~/.config/opencode/config.json` not yet configured for M5 Gemma4 provider.