MarkBase架构升级:Multi-Volume Virtual Tree + Dual-View Management + Git Remote修正
核心功能: - ✅ Categories/Series双视图管理(category_view.rs + import_markdown.rs) - ✅ FUSE Multi-Volume支持(tree_type参数) - ✅ SSH/SFTP/SCP/rsync协议完整实现(4042行) - ✅ NFS/SMB Module Phase 1-3完成 - ✅ Archive Module Phase 1-4完成(2916行) - ✅ Download Center API完整实现 - ✅ S3兼容API实现(560行) Git配置修正: - ✅ 删除错误origin(gitea.momentry.ddns.net) - ✅ 删除m5max128(指向机器名) - ✅ 设置origin = m5max128gitea.momentry.ddns.net/admin/markbase - ✅ 设置m4minigitea = m4minigitea.momentry.ddns.net/warren/markbase 数据清理: - ✅ 删除38个临时SQLite(保留accusys.sqlite、demo.sqlite) - ✅ 删除.bak、test_*.bin、调试脚本等临时文件 - ✅ 删除临时目录(build/、download files/、raid_test/等) - ✅ 更新.gitignore排除临时文件 架构优化: - 52个文件修改,2434行新增,4739行删除 - Workspace成员整合(16个crate) - 数据库状态:accusys.sqlite保留(主demo测试) 远程同步: - ✅ 准备推送到m5max128gitea(远程Gitea) - ✅ 准备推送到m4minigitea(本地Gitea)
This commit is contained in:
175
docs/HYBRID_IMPLEMENTATION_GUIDE.md
Normal file
175
docs/HYBRID_IMPLEMENTATION_GUIDE.md
Normal file
@@ -0,0 +1,175 @@
|
||||
# SQLite + Sled 混合架构实施指南
|
||||
|
||||
**实施日期:** 2026-05-29
|
||||
**实施目标:** 6天POC实施 + 生产部署评估
|
||||
|
||||
---
|
||||
|
||||
## 实施路线图
|
||||
|
||||
### Phase 1: 基础架构 (Day 1-2)
|
||||
|
||||
```
|
||||
Day 1: HybridRouter 实现
|
||||
├── HybridRouter基础框架
|
||||
├── 数据路由逻辑
|
||||
└── 基础查询API
|
||||
|
||||
Day 2: 缓存机制实现
|
||||
├── metadata_cache Tree
|
||||
├── 缓存查询逻辑
|
||||
└── 缓存失效机制
|
||||
```
|
||||
|
||||
### Phase 2: 数据同步 (Day 3-4)
|
||||
|
||||
```
|
||||
Day 3: 双写机制实现
|
||||
├── 双写同步逻辑
|
||||
├── 一致性检查
|
||||
└── 自动修复机制
|
||||
|
||||
Day 4: 缓存优化实现
|
||||
├── hot_files_cache Tree
|
||||
├── LRU淘汰机制
|
||||
└── TTL过期清理
|
||||
```
|
||||
|
||||
### Phase 3: 性能验证 (Day 5-6)
|
||||
|
||||
```
|
||||
Day 5: 性能测试
|
||||
├── 混合架构性能测试
|
||||
├── 缓存命中率测试
|
||||
└── 并发性能测试
|
||||
|
||||
Day 6: 优化调优
|
||||
├── 缓存配置优化
|
||||
├── 并发配置优化
|
||||
└── 性能对比验证
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 核心代码框架
|
||||
|
||||
### HybridRouter 核心
|
||||
|
||||
```rust
|
||||
pub struct HybridRouter {
|
||||
sqlite_conn: Connection,
|
||||
sled_db: sled::Db,
|
||||
}
|
||||
|
||||
impl HybridRouter {
|
||||
pub fn route_query(&self, query_type: QueryType) -> DatabaseType {
|
||||
match query_type {
|
||||
// SQL查询 → SQLite
|
||||
QueryType::ParentChildren => DatabaseType::SQLite,
|
||||
QueryType::FileUuidJoin => DatabaseType::SQLite,
|
||||
QueryType::WhereFilter => DatabaseType::SQLite,
|
||||
|
||||
// KV查询 → Sled
|
||||
QueryType::ContentHashLookup => DatabaseType::Sled,
|
||||
QueryType::HotFileCache => DatabaseType::Sled,
|
||||
QueryType::MetadataCache => DatabaseType::Sled,
|
||||
|
||||
// 混合查询 → 优先缓存
|
||||
QueryType::NodeLookup => DatabaseType::Hybrid,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn get_node(&self, node_id: &str) -> Result<Option<FileNode>> {
|
||||
// 混合策略:
|
||||
// 1. Check Sled cache (fast)
|
||||
// 2. If not found, query SQLite (slow)
|
||||
// 3. Update Sled cache
|
||||
|
||||
let cache_tree = self.sled_db.open_tree("metadata_cache")?;
|
||||
|
||||
// Step 1: Check cache
|
||||
if let Some(cache_data) = cache_tree.get(node_id.as_bytes())? {
|
||||
let cache: CachedMetadata = serde_json::from_slice(&cache_data)?;
|
||||
if cache.ttl > 0 {
|
||||
return Ok(Some(cache.to_file_node()));
|
||||
}
|
||||
}
|
||||
|
||||
// Step 2: Query SQLite
|
||||
let node = self.sqlite_query_node(node_id)?;
|
||||
|
||||
// Step 3: Update cache
|
||||
if let Some(n) = &node {
|
||||
let cache = CachedMetadata::from_node(n);
|
||||
cache_tree.insert(node_id.as_bytes(), serde_json::to_vec(&cache)?)?;
|
||||
}
|
||||
|
||||
Ok(node)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 快速启动命令
|
||||
|
||||
### POC 测试
|
||||
|
||||
```bash
|
||||
# 1. 创建混合架构模块
|
||||
mkdir -p filetree-hybrid/src
|
||||
|
||||
# 2. 编译运行
|
||||
cargo build --release --package filetree-hybrid
|
||||
cargo run --release --bin hybrid-poc-test
|
||||
|
||||
# 3. 性能对比测试
|
||||
cargo run --release --bin hybrid-benchmark
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 监控指标
|
||||
|
||||
### 缓存命中率目标
|
||||
|
||||
```
|
||||
Target Cache Hit Rate:
|
||||
┌─────────────────────────────────┐
|
||||
│ Level 5: FUSE hot path │ → 95%+ hit rate
|
||||
│ Level 4: Frequently accessed │ → 90%+ hit rate
|
||||
│ Level 3: Normal accessed │ → 80%+ hit rate
|
||||
│ Level 2: Rarely accessed │ → 60%+ hit rate
|
||||
│ Level 1: Cold files │ → 40%+ hit rate
|
||||
│ Level 0: Archive files │ → 20%+ hit rate
|
||||
└─────────────────────────────────┘
|
||||
|
||||
Overall Target: 85%+ cache hit rate
|
||||
```
|
||||
|
||||
### 性能目标
|
||||
|
||||
```
|
||||
Performance Targets:
|
||||
├── Query latency: <1ms (cache hit) / <5ms (cache miss)
|
||||
├── Import throughput: >100K/sec (Sled batch)
|
||||
├── Concurrent reads: >1M/sec (Sled MVCC)
|
||||
└── Cache hit rate: >85%
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 部署决策矩阵
|
||||
|
||||
| 触发条件 | 当前状态 | 阈值 | 决策 |
|
||||
|----------|----------|------|------|
|
||||
| **并发用户** | 1-3 | >10 | 开始POC |
|
||||
| **导入吞吐** | 14K/sec | >50K/sec | 开始POC |
|
||||
| **查询延迟** | <1ms | >10ms | 开始优化 |
|
||||
| **缓存需求** | 无 | >80%命中率 | 开始POC |
|
||||
|
||||
---
|
||||
|
||||
## 一句话总结
|
||||
|
||||
**6天POC实施,保留SQLite SQL优势 + 利用Sled性能优势,达到85%+缓存命中率。**
|
||||
Reference in New Issue
Block a user