1. Add Redis PipelineProgress cleanup in unregister_internal
- Deletes progress key when file is unregistered
2. Fix ingestion_complete asr_status query
- Query ASRX first, then ASR
- Filter out NULL values to avoid false negatives
3. Fix TKG 0 nodes/edges support
- TKG is considered done if face traces are complete
- TKG may create 0 nodes/edges for videos with minimal content
4. Add ASRX fallback to ASR segments
- When ASRX has 0 segments but ASR has segments, use ASR segments
- Ensures at least one ASRX output when ASR has content
When a job completes via the essential_completed branch (all essential
processors done but some non-essential failed), the PipelineProgress
was not being updated to 100%, causing a discrepancy where the status
showed 'completed' but progress showed <100%.
Now PipelineProgress.mark_completed() is called in both completion paths.
When ASRX falls back to ASR segments, end_frame may be 0.
The FPS calculation now handles this case correctly by checking
both end_frame > 0 and end_time > 0 before dividing.
This prevents division by zero and incorrect FPS values when
processing videos with ASRX fallback segments.
1. trace_done now checks for 'no_faces' status in face_traced.json
- Videos with no detected faces now complete correctly
- Previously stuck because trace_count=0 returned false
2. ASRX fallback to ASR segments includes start_frame/end_frame
- Added _convert_asr_segments_to_asrx helper function
- TKG can now process fallback segments correctly
This allows processing of:
- Videos with no audio track (ASR: no_audio_track)
- Videos with no faces (face_traced.json: no_faces)
When a file is re-registered, the old PipelineProgress in Redis
was not cleared, causing the Portal to show 'completed' status
even though the new job was still running.
Now PipelineProgress is deleted when a new job starts processing.
Previously, ingestion_complete only checked ASR and face traces,
causing jobs to complete before TKG was triggered. This resulted in
TKG nodes being 0 even after job completion.
Now ingestion_complete also checks if TKG nodes exist for the file.
The job stays in 'running' status until TKG completes.
The TKG guard was checking if nodes exist before spawning TKG build.
This caused a deadlock when:
1. TKG spawns (async)
2. ingestion_complete checks → false (nodes not yet created)
3. Returns Ok(false), job stays running
4. Next poll: guard checks nodes → no nodes yet (TKG still running)
5. Skips TKG → deadlock!
Since build_tkg uses ON CONFLICT (idempotent), it's safe to call
multiple times. Removed the guard to fix the deadlock.
When a job is deleted from the database (e.g., by unregister),
the worker now checks if the job still exists before processing.
If the job no longer exists, it skips it instead of getting stuck.
This fixes the issue where unregistering a file would leave the
worker stuck trying to process a non-existent job.
Identity agent now checks Qdrant _seeds collection for the file's
seed identity photos before running. If no seeds exist, the agent
is skipped to avoid unnecessary processing.
Flow:
1. Job completes (Face + ASRX done)
2. Check Qdrant _seeds for file_uuid
3. If seeds exist → run identity agent
4. If no seeds → skip identity agent
Previously Rule 1 only created chunks from ASRX segments, merging OCR
text where frame ranges overlapped. OCR text that didn't overlap with
any ASRX segment was ignored.
Now Rule 1 has two phases:
1. Process ASRX segments (merge OCR where overlapping) - existing behavior
2. Create chunks for OCR-only text (frames not covered by ASRX)
OCR-only chunks are grouped by consecutive frames (within 5 frames)
to avoid creating too many single-frame chunks.
Example: ASRX 819 + OCR-only 4 = 823 sentence chunks
Bug: ASRX handler stored pre_chunks as BOTH 'asrx' and 'asr' types.
This caused confusion because Rule 1 queries 'asrx' type, but only
'asr' type existed in the database (asrx type was deleted or never stored).
Fix: Remove the duplicate 'asr' storage (lines 530-542).
ASRX handler now only stores 'asrx' type pre_chunks to workspace SQLite.
PostgreSQL pre_chunks are stored by processor.rs with correct 'asrx' type.
This ensures Rule 1 can find ASRX pre_chunks correctly.
Frontend:
- Add media type filter (全部/影片/照片)
- Add indexed status filter (未入庫/已入庫)
- Show media type column with icons
- Fix status filter to handle indexed/unindexed correctly
- Determine media type from file extension
Backend:
- Add total_chunks field to FileItem API response
- Query chunk counts efficiently in batch with IN clause
- Frontend uses total_chunks to determine is_indexed status
Bug: all_completed only checked existing results, not missing processors.
If a processor (like pose) never created a result row, all_completed would
still return true and mark the job as completed.
Fix: all_completed now checks that every processor in job_processors has
a corresponding completed result. Added logging for missing processors.
Also fixed:
- any_pending now checks all expected processors, not just existing results
- Added missing_processors detection and logging
- Skip chunks where both ASRX text and OCR text are empty
- Use count-based chunk_id instead of index to avoid gaps
- This ensures PostgreSQL and Qdrant chunk counts match
- Added text_content field to SearchResult and SemanticSearchResult
- Added get_chunk_by_id_no_embedding for keyword results without embedding requirement
- Fixed search_bm25 to use position-based ranking for CJK/Korean content
- Fixed sqlx column mapping with explicit alias
- Skip text_match filter for keyword-only results
- Use text_content as fallback when summary is empty
- Frontend calls /api/v1/face-thumbnail?uuid=...&frame=...
- Backend only had /api/v1/file/:file_uuid/thumbnail
- Added compat route and uuid field to ThumbQuery
- ingestion_complete query used file_uuid column which is always NULL
- Changed to JOIN processor_results with monitor_jobs on job_id
- All stuck jobs now complete successfully
- Removed trace_chunks field from PostgresStats struct
- Removed trace_chunks query from get_file_stats and get_ingestion_status
- Fixed OCR fetch_ocr_texts to compute frames from start_time*FPS
- Updated scan.rs to use separate count_nodes/count_edges functions
- get_pipeline_progress_handler now queries actual DB counts
- Fixed processor_results query (requires JOIN with monitor_jobs)
- Card progress bar and right-click content now consistent
- tkg_nodes has no edge_type column, query was failing silently
- Split into count_nodes(node_type) and count_edges(edge_type)
- Fixed text_region → text_trace node type name
- Also: OCR frame fix in rule1 (end_frame computed from end_time+FPS)
- get_file_identities: UNION face_detections + file_identities
- list_identities: add file_bindings from file_identities table
- Add back /api/v1/traces/unassigned route
- Total count query now includes file_identities
Frontend can now:
- Filter pending identities by file_uuid
- Filter pending faces (unassigned traces) by file_uuid
- skin_tone is a person attribute (like height), not trace attribute
- Remove build_skin_tone_trace_nodes function
- Remove skin_tone_trace_nodes from TkgResult and API response
- Remove skin_tone_trace from documentation tables
- Fix trace_id type mismatch (INT4 vs i64) with explicit ::bigint cast
- Change build_face_track_nodes to use from_pg version
- Add skin_tone_trace_nodes to API response
- Add #[derive(Serialize)] to TkgResult
- Fix Unicode panic in text label truncation
- Add push_existing_embeddings.py script
- Add public health routes at /api/v1/health, /api/v1/health/detailed, /api/v1/health/consistency
- Make health functions and response types public
- Public routes bypass auth middleware (unlike protected /api/v1/* routes)
- Add Queued variant to VideoStatus enum
- Trigger sets videos.status='queued' instead of staying 'pending'
- Worker sets videos.status='processing' on pickup
- list_monitor_jobs_by_status ORDER BY created_at ASC (FIFO)
- queue_position counts both 'pending' and 'queued' jobs
- Identity agent: per-face max matching, multi-round with derived
seeds from high-confidence faces, angle diversity filter (cosine sim < 0.90)
- Pending person API: POST /file/:file_uuid/pending-person
+ GET /file/:file_uuid/pending-persons with status=pending, source=manual
- Update API docs (07_identity.md)
- Remove Rule 3 (Scene Chunking) from worker auto-trigger
- Remove rule3_ingest.rs and related imports
- Remove Story/Caption from playground module parsing
- Clean up scan.rs Rule 3 display
- Fix ASRX field name conversion (start_time -> start)
Reason: Story/5W1H/Scene accuracy too poor - will redesign later
- Problem: compact=p=0:nk=1 outputs pipe-delimited format without pts_time=
- Fix: default=nk=0 outputs pts_time=XXX format that parser can match
- Result: Charade scene detection from 1 scene -> 833 scenes (correct)
- pre_chunks: add chunk_type, text_content columns; drop NOT NULL on
coordinate_type/coordinate_index (INSERT statements reference these
columns but CREATE TABLE was missing them)
- run_migrations: add ALTER TABLE for existing databases
- extract_movie_name: filter noise words (youtube, fps, 24fps, 1080p,
pure digits) so 'Charade_YouTube_24fps' → 'Charade'
- run-server-3002.sh: add companion worker startup (matching 3003 script)
- ProcessorType::all(): remove MediaPipe, Appearance, Story (mediapipe replaced by Swift)
- files.rs auto-pipeline: fix order to cut,asr,asrx,yolo,ocr,face,pose (was missing asr)
- postgres_db.rs run_migrations(): rewrite to auto-create all 38 tables idempotently
