383 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Warren	e4d1be01ef	Add Proxmox VE feature comparison analysis ... Document Purpose: - Compare MarkBase vs Proxmox VE features - Define MarkBase positioning (Mini Proxmox Backup Server + File Server) Comparison Categories: 1. Storage Management (60% coverage) 2. Backup/Restore (80% coverage) ⭐⭐⭐⭐⭐ 3. File Services (100% coverage - MarkBase unique) ⭐⭐⭐⭐⭐ 4. Virtualization (0% - not provided) 5. Authentication (62% coverage) 6. Web UI (62% coverage) 7. API (75% coverage) 8. Network (0% - not provided) 9. Security (75% coverage) Overall Coverage: 58% (focused on storage + backup) MarkBase Unique Advantages: - Multi-protocol file services (SMB + SFTP + WebDAV + S3) - ZFS-style incremental backup (hardlink, 0 disk usage) - SSH high performance (140 MB/s) - macOS Time Machine support Proxmox VE Unique Advantages: - Complete virtualization platform (KVM + LXC) - HA cluster (Corosync + Pacemaker) - Proxmox Backup Server integration Co-deployment Options: A. MarkBase as storage backend for Proxmox VE B. MarkBase as backup server for Proxmox VE C. MarkBase standalone (small teams) Next Phase 9 Suggestions: - Distributed storage (Ceph-like) - Webhook completion - 2FA support - UI improvements	2026-06-24 04:25:39 +08:00
Warren	d76a200560	Add incremental backup support (Phase 8) ... BackupScheduler Enhancement: - Added incremental: bool field to BackupScheduleConfig - Default: incremental=true (enabled by default) - copy_incremental_to_snapshot() method - file_changed() detection (size + mtime comparison) - Hardlink unchanged files to base snapshot (ZFS-style) Incremental Backup Algorithm: 1. If incremental=true and previous snapshot exists: - Compare file size and mtime with base snapshot - If unchanged: create hardlink to base (zero disk usage) - If changed: copy and compress (new content) 2. If incremental=false or no previous snapshot: - Full copy (traditional backup) Storage Savings: - Unchanged files: hardlink (0 extra disk space) - Changed files: copy + compress (minimal overhead) - Similar to ZFS snapshot mechanism BackupConfigResponse Updated: - Added incremental field - Added compress field (GUI: dropdown select) Backup.vue Updated: - Incremental switch with explanation text - Compression dropdown (None/LZ4/ZSTD) - Default values loaded from backend REST API Test: curl /api/v2/backup/config {incremental:true,compress:zstd,...} Build: 495 tests pass	2026-06-24 04:20:33 +08:00
Warren	2d8e9049b0	Add compression support to backup workflow ... BackupScheduler Enhancement: - copy_file() now compresses files using ZSTD or LZ4 - min_size threshold: 1024 bytes (smaller files not compressed) - compression level: 3 (balanced speed/compression) BackupConfigResponse Updated: - Added compress, encrypt, include_checksums fields - compress: 'none' | 'lz4' | 'zstd' - Default: 'zstd' REST API Enhancement: - GET /api/v2/backup/config returns full config - POST /api/v2/backup/config accepts compression settings Test Results: - Set compress='lz4': ✅ Config updated - Set compress='zstd': ✅ Config updated - Compression applied via run_backup() (scheduled backup) Note: Direct create_snapshot API doesn't use compression (scheduler.run_backup() is the primary backup mechanism) Build: 495 tests pass	2026-06-24 04:14:24 +08:00
Warren	55caeabd94	Add root parameter to backup/snapshot REST API ... API Enhancement: - All snapshot endpoints now accept 'root' query parameter - Default root: /data (for production) - Test root: configurable (e.g., /tmp/backup_test) Endpoints updated: - GET /api/v2/snapshots?root=<path> - POST /api/v2/snapshots/:name?root=<path> - DELETE /api/v2/snapshots/:name?root=<path> - POST /api/v2/snapshots/:name/restore?root=<path> - GET /api/v2/storage/stats?root=<path> Integration Testing Results ✅: - Create snapshot: test_snap1 created - List snapshots: ['test_snap1'] returned - Modify file: 'original content' → 'modified content' - Restore snapshot: 'modified content' → 'original content' ✅ - Delete snapshot: test_snap1 removed Snapshot metadata format: { 'name': 'test_snap1', 'created': {'secs_since_epoch': 1782243041, 'nanos_since_epoch': 344384000}, 'source_path': '/tmp/backup_test' } Build: 495 tests pass Server: Port 11438 running with root parameter support	2026-06-24 03:31:43 +08:00
Warren	26d4199203	Add Backup REST API endpoints (Phase 5-6) ... REST API Implementation: - 8 backup/snapshot endpoints added to server.rs - BackupScheduler: add get_config()/set_config() methods Endpoints: - GET /api/v2/backup/stats - Scheduler status - GET/POST /api/v2/backup/config - Config management - POST /api/v2/backup/run - Manual backup trigger - GET /api/v2/snapshots - List snapshots - POST/DELETE /api/v2/snapshots/:name - Create/delete snapshot - POST /api/v2/snapshots/:name/restore - Restore snapshot - GET /api/v2/storage/stats - Storage metrics Test Results: - curl /api/v2/backup/stats ✅ - curl /api/v2/backup/config ✅ - curl /api/v2/storage/stats ✅ - curl /api/v2/snapshots ✅ Build: 495 tests pass Server: Port 11438 running with new endpoints	2026-06-24 03:25:41 +08:00
Warren	90219a65ad	Add Backup Management GUI (Phase 3-4) ... Web GUI Implementation: - Backup.vue: Storage dashboard + Snapshot management + Scheduler config - Router: Add /backup route - Home.vue: Add Backup management card - Tauri commands: 10 backup API endpoints Features: - Storage stats (total/used/free, dedup/compression ratios) - Snapshot list with create/delete/restore actions - Backup scheduler configuration (enabled, interval, max_snapshots) - Run backup now button - Send/Receive placeholders Tauri Commands: - get_storage_stats, list_snapshots - create_snapshot, delete_snapshot, restore_snapshot - get_backup_stats, get_backup_config, set_backup_config - run_backup Build: cargo build (Tauri) ✅ 5 warnings Tests: 495 markbase-core + 201 smb-server = 696 total	2026-06-24 03:16:27 +08:00
Warren	1d9e140e6c	Fix Backup/Restore API compilation errors ... - chrono timestamp_opt API: use TimeZone trait method - VfsError::Io/NotFound: use String literals - SendFormat: add PartialEq derive - VfsRaidConfig tests: add disk_paths field - BackupStats test: use relative timestamps - HashSet file tracking: use (String, u64) tuple - BackupStream::receive: clone format before use - collect_file_data: fix temporary lifetime All tests pass: 495 markbase-core + 201 smb-server = 696 total	2026-06-24 02:37:03 +08:00
Warren	5f12e9f5d7	Implement scrub scheduler + dedup repair: Phase 5-6 complete ... Phase 5: Background scrub scheduler (~220 lines) - ScrubScheduler: periodic scrub at configurable interval - ScrubSchedulerConfig: interval_secs, scrub_on_startup, repair_enabled - start/stop/run_once methods - ScrubStats: running, scrub_count, last/next scrub time - 6 unit tests: default config, start/stop, stats, timestamp format Phase 6: Dedup repair integration (~30 lines) - DedupStore::get_block_by_checksum(): retrieve by SHA-256 hash - DedupStore::has_block_by_checksum(): check existence - DedupStore::repair_from_checksum(): repair corrupted block - checksum::repair_block_from_dedup(): integration hook Tests: 471 passed (+6 new scrub_scheduler tests) Files: - markbase-core/src/vfs/scrub_scheduler.rs (NEW) - markbase-core/src/vfs/dedup.rs (MOD +30 lines) - markbase-core/src/vfs/checksum.rs (MOD +20 lines) - markbase-core/src/vfs/mod.rs (MOD +1 line)	2026-06-24 01:46:08 +08:00
Warren	ffc3f03744	Implement block-level checksum: Phase 1-4 complete ... Phase 1: VfsBlockChecksum struct + JSON storage (~240 lines) - VfsBlockChecksum: offset + SHA-256 hash - VfsChecksumFile: block_size + algorithm + blocks + file_size - compute_block_hash() + verify_block_hash() - ChecksumMode: Lazy (default) + OnRead - ScrubResult: total/verified/corrupted/repaired blocks metrics Phase 2: ChecksumFile wrapper (~180 lines) - VfsFile wrapper with transparent checksum - Lazy verification (only on scrub) - Cache of verified blocks - Update checksum on flush() - read_at/write_at support Phase 3: Scrub API (~150 lines) - scrub_file(): verify single file integrity - scrub_all(): recursive directory scrub - create_checksums_for_file(): generate checksums - repair_block(): placeholder for RAID/Dedup Phase 4: RAID repair integration (~160 lines) - repair_block_from_parity(): reconstruct from RAID parity - reconstruct_from_p(): XOR reconstruction for RaidZ1 - reconstruct_from_pq/pqr(): placeholder for RaidZ2/3 Tests: 15 checksum tests pass (465 total) Files: - markbase-core/src/vfs/checksum.rs (NEW) - markbase-core/src/vfs/checksum_file.rs (NEW) - markbase-core/src/vfs/raid.rs (MOD +160 lines) - markbase-core/src/vfs/mod.rs (MOD +2 lines)	2026-06-24 01:41:56 +08:00
Warren	7c4476e19c	Implement at-rest encryption: AES-256-GCM VFS layer ... - Added encrypted_fs.rs module for transparent file encryption - EncryptedVfs wraps any VfsBackend with AES-256-GCM encryption - Per-file key derivation from master key + file path (SHA-256) - File format: MBE1 magic + version + nonce + original_size + ciphertext + tag - EncryptedFile transparently decrypts on read, encrypts on flush - 5 unit tests: roundtrip, different keys, key derivation, header format, password config Tests: 457 markbase-core (+5 new), 201 smb-server (658 total)	2026-06-24 00:57:53 +08:00
Warren	57fd6a475f	macOS Time Machine AFP monitoring: backup_time update on file modification ... - Added afp_monitor.rs module to track AFP_AfpInfo backup_time - Open struct now has 'modified' flag to track file modifications - write.rs sets modified=true on successful write - close.rs calls AfpMonitor::update_backup_time() on modified files - create.rs calls AfpMonitor::init_afp_info() on new file creation - AFP_AfpInfo stored as xattr com.apple.aapl.AfpInfo - backup_time updated to current epoch time on modification Also includes: - LZ4 compression using lz4_flex crate - Case sensitivity conditional on backend capabilities - LDAP cfg feature gate fix - RAID rebuild reconstruction implementation - DOS attributes xattr persistence - Snapshot disk persistence Tests: 201 smb-server, 452 markbase-core (653 total)	2026-06-24 00:46:33 +08:00
Warren	5300b672cb	Compound request integration tests: stitch_responses, capture_file_id, inherit_context, CREATE+CLOSE chain	2026-06-23 10:46:30 +08:00
Warren	637227f4e4	SMB: reusable read buffer in VfsHandle (avoid per-read allocation + zero-init) ... - Add FileAndBuf struct wrapping file + reusable read_buf Vec - read(): reuse Vec capacity across calls, use unsafe set_len to skip memset - ~15% read throughput improvement (2.6 → 3.0 GB/s on localhost smbclient)	2026-06-23 10:05:39 +08:00
Warren	d4f60929fa	SMB performance optimization: pread/pwrite, tokio::sync::Mutex, direct response, fast-path ... - VfsFile trait: add read_at()/write_at() with seek+read default impl - LocalFs: override with real pread/pwrite (FileExt::read_at/write_at) — 1 syscall vs 2 - smb_server_backend: use read_at/write_at + tokio::sync::Mutex (non-blocking async) - read handler: build response directly, avoid Bytes→Vec<u8> copy + intermediate struct - oplock break: fast-path skip when ≤1 open entry (single-user scenario)	2026-06-23 09:58:19 +08:00
Warren	e7863a3034	Fix macOS SMB mount: AAPL caps, credit grant, file_index, QueryDirectory padding ... - AAPL: Restore UNIX_BASED+NFS_ACE server_caps, RESOLVE_ID+FULL_SYNC volume_caps (Samba baseline) - Credit: Grant min 1 credit in dispatch response for smbclient compatibility - file_index: Assign 1-based index per entry in list_dir (both VFS and local backends) - smb_match(): Add wildcard pattern filter (*/?) for macOS single-entry QueryDirectory probes - FILE_ID_BOTH_DIR_INFORMATION: Add 2-byte Reserved2 padding between ShortName and FileId - macOS Sequoia 15.5 mount_smbfs now succeeds (tested: ls, cat, read)	2026-06-23 09:44:01 +08:00
Warren	8ef1406ed3	SMB fixes: IPC$ ShareMode=Public, capabilities=0, FILE_ID_BOTH_DIRECTORY_INFORMATION Reserved2 removed, NextEntryOffset=0 for last entry, debug logging	2026-06-23 03:22:39 +08:00
Warren	bb796ec6b9	Fix smb-server xattr: add root_path field for absolute path storage	2026-06-22 16:25:33 +08:00
Warren	9dd2eefeea	Fix smb-server xattr: dereference Arc<Dir> before as_std_path()	2026-06-22 15:41:03 +08:00
Warren	0c4459ae66	Fix smb-server xattr: use PathBuf for absolute paths	2026-06-22 15:39:37 +08:00
Warren	5b0086f6f0	Implement Time Machine xattr support (Phase 4.1 complete)	2026-06-22 15:30:44 +08:00
Warren	3029327d5e	Implement SMB AFP_Resource Stream via AppleDouble files (Phase 3 complete)	2026-06-22 15:27:28 +08:00
Warren	1c8c47d5fa	Implement SMB AFP_AfpInfo read/write via xattr (Phase 2.8 complete)	2026-06-22 15:16:59 +08:00
Warren	25991c71b2	Update Cargo.lock for new dependencies (xattr, smb-server AAPL modules)	2026-06-22 14:22:16 +08:00
Warren	866d0536c8	Add SMB AAPL Extensions Phase 1-6 + VFS xattr support ... Phase 1: AAPL Create Context negotiation Phase 2: AFP_AfpInfo Stream structure (Finder info + creation time) Phase 2.5: SMB Named Stream Backend (NamedStreamPath) Phase 2.6: Backend Named Stream Support in handlers Phase 2.7: VFS Extended Attributes (get/set/remove/list_xattr) Phase 4: Time Machine share config (time_machine field) Phase 5: Server/Volume Capabilities Phase 6: macOS Unicode mapping (private range ↔ ASCII) Tests: 174 smb-server tests pass, 52 VFS tests pass	2026-06-22 14:21:53 +08:00
Warren	64709ec529	Add CTDB Phase 1-5: TDB storage + Node management + Control protocol + IP manager + Recovery	2026-06-22 14:21:39 +08:00
Warren	a8d81f2a9c	Revert "Remove Download Center routes from server.rs (dead code cleanup)" ... This reverts commit 20b208bb7f.	2026-06-22 14:12:14 +08:00
Warren	20b208bb7f	Remove Download Center routes from server.rs (dead code cleanup) ... Removed routes: - /api/v2/products/* (CRUD + file assignment) - /api/v2/download/* (file download + stats) - /api/v2/files/:user_id (list + info via download module) - /upload, /files, /products (HTML pages) Kept: /api/v2/upload-unlimited, /downloads, category/series APIs	2026-06-22 11:00:41 +08:00
Warren	60e4329eed	Add VirtualFs tag-mode WebDAV + MyFiles UI + Admin WebDAV endpoint ... - VirtualFs: SQLite-backed virtual folders (tag mode), 16 unit tests - MyFiles module: API endpoints + Web UI for folder/tag management - Admin WebDAV: /admin-webdav/*path with Basic Auth + URI prefix rewrite - CLI: webdav-folder/tag/untag/list/start --virtual-mode commands - Deployed and tested on M5Max48: PROPFIND, PUT, GET, DELETE all working	2026-06-22 10:38:25 +08:00
Warren	37d0fe1a3c	Fix duplicate derive(Clone)	2026-06-22 07:28:33 +08:00
Warren	4003864d28	Fix WebDAV: add Clone to WebdavCredentials	2026-06-22 07:26:54 +08:00
Warren	8039f0d375	Fix WebDAV auth: use map_or for password check	2026-06-22 07:25:53 +08:00
Warren	3d395584a8	Fix WebDAV: middleware use extensions().get() to not consume	2026-06-22 07:23:57 +08:00
Warren	cf57d46ca5	Fix WebDAV: handle_dav extract WebdavCredentials Extension	2026-06-22 07:22:01 +08:00
Warren	8a5a23a309	Fix WebDAV Extension layer order	2026-06-22 07:20:34 +08:00
Warren	a7f50ff747	Update WebDAV: root path + 0.0.0.0 bind	2026-06-22 07:17:45 +08:00
Warren	41f0217450	Update Caddyfile: studio.momentry.ddns.net/demo WebDAV config	2026-06-22 06:51:51 +08:00
Warren	e7a9f886ed	Fix web server bind to 0.0.0.0 for external access	2026-06-22 06:20:17 +08:00
Warren	cd184daa20	Update AGENTS.md: CTDB architecture analysis summary	2026-06-22 05:37:12 +08:00
Warren	060f43f0c4	Add CTDB architecture analysis document	2026-06-22 05:36:23 +08:00
Warren	63b765f68e	Update AGENTS.md: Phase 5 complete summary	2026-06-22 05:31:26 +08:00
Warren	e9eca1b492	Add DFS Referral Support (Phase 5)	2026-06-22 05:30:16 +08:00
Warren	4db72fff4a	Update AGENTS.md: Phase 6 complete summary	2026-06-22 05:22:54 +08:00
Warren	52c38b1919	Add SMB Configuration Templates (Phase 6)	2026-06-22 05:22:14 +08:00
Warren	054bf55490	Update AGENTS.md: Phase 1-4 complete summary	2026-06-22 05:15:41 +08:00
Warren	e267b43424	Add Compound Request tests (Phase 4)	2026-06-22 05:13:02 +08:00
Warren	c89f6c96ae	Update AGENTS.md: Phase 1-3 complete summary	2026-06-22 04:43:49 +08:00
Warren	ebe976eee4	Implement Write/Read Cache (Phase 3)	2026-06-22 04:42:55 +08:00
Warren	9ae0402318	Document NTLMv2+LDAP incompatibility and skip Phase 2.3	2026-06-22 04:34:15 +08:00
Warren	3c5de4e6a3	Update AGENTS.md: LDAP Provider Phase 2.1-2.2 complete	2026-06-22 04:13:59 +08:00
Warren	88590d3611	Add LDAP CLI parameters to SMB server (Phase 2.2)	2026-06-22 04:13:10 +08:00