markbaseengine/支持模型列表.md

# MarkBase-12B 支持的模型列表

## 模型架构支持

### 当前支持的模型类型

**Gemma-4 系列**:
- ✓ Gemma-4 E4B (Early Access 4B)
- ✓ Gemma-4 12B
- ✓ E4B-MarkBase (multimodal variant)
- ✓ MarkBase-12B (multimodal variant)

**模型架构**:
- Gemma4ForConditionalGeneration (multimodal)
- 42层 Transformer
- 262,144 vocabulary size
- Vision Tower (16 layers)
- Audio Tower (12 layers)

---

## 模型加载方式

### 1. 从本地目录加载

**基本用法**:
```bash
swift run G12BServer <model_dir> <port> <model_id>
```

**示例**:
```bash
# 加载 E4B-MarkBase
swift run G12BServer /Users/accusys/MarkBase12B/models/E4B-MarkBase 8080 markbase-12b

# 加载其他模型
swift run G12BServer /path/to/your/model 8080 custom-model
```

**所需文件**:
```
model_dir/
  model.safetensors - 模型权重（4-bit quantized）
  model.safetensors.index.json - 权重索引（如果分片）
  config.json - 模型配置
  tokenizer.json - Tokenizer
  tokenizer_config.json - Tokenizer配置
  generation_config.json - 生成配置
```

### 2. Safetensors 格式要求

**权重格式**:
- Safetensors binary format
- 4-bit quantization (uint32 packed)
- Group size: 64
- BF16 scales/biases

**支持的数据类型**:
- INT4 (4-bit quantized)
- BF16 (scales/biases)
- F32 (activations)

### 3. 配置文件格式

**config.json 示例**:
```json
{
  "model_type": "gemma4",
  "architectures": ["Gemma4ForConditionalGeneration"],
  "hidden_size": 2560,
  "num_hidden_layers": 42,
  "vocab_size": 262144,
  "num_attention_heads": 8,
  "num_key_value_heads": 2,
  "head_dim": 256,
  "intermediate_size": 10240,
  "max_position_embeddings": 131072,
  "quantization_config": {
    "bits": 4,
    "group_size": 64
  }
}
```

**Vision 配置**:
```json
{
  "vision_config": {
    "hidden_size": 768,
    "num_hidden_layers": 16,
    "patch_size": 16,
    "image_size": 224
  }
}
```

**Audio 配置**:
```json
{
  "audio_config": {
    "hidden_size": 640,
    "num_hidden_layers": 12,
    "num_mel_bins": 128
  }
}
```

---

## 模型选择指南

### 根据 Use Case 选择

#### 1. 纯文本推理
**推荐模型**: Gemma-4-4B-IT (Instruction Tuned)
- 适用: 文本生成、对话、问答
- 不需要: Vision/Audio components
- 性能: 更快（无 multimodal overhead）

#### 2. 视觉理解
**推荐模型**: E4B-MarkBase, MarkBase-12B
- 适用: 图像描述、视觉问答、场景理解
- 需要: Vision Tower (16 layers)
- 输入: 224x224 RGB images

#### 3. 音频理解
**推荐模型**: MarkBase-12B (with audio tower)
- 适用: 音频描述、语音识别、音频问答
- 需要: Audio Tower (12 layers)
- 输入: Mel spectrograms (128 bands)

#### 4. 多模态推理
**推荐模型**: E4B-MarkBase, MarkBase-12B
- 适用: Vision + Audio + Text
- 需要: Vision + Audio Towers
- 输入: Images + Audio + Text

#### 5. 分布式推理
**推荐模型**: 12B variants
- 适用: 跨设备推理、高性能
- 需要: RDMA setup (Thunderbolt 5)
- 性能: 658 tok/s (distributed)

---

## 模型规格对比

### Gemma-4 模型系列

| 模型 | 参数量 | Layers | Hidden Size | Vocab | Vision | Audio |
|------|--------|--------|-------------|-------|--------|-------|
| Gemma-4-4B | 4B | 42 | 2560 | 262K | - | - |
| Gemma-4-12B | 12B | 42 | 3072 | 262K | - | - |
| E4B-MarkBase | ~12B | 42 | 2560 | 262K | ✓ (16) | ✓ (12) |
| MarkBase-12B | ~12B | 42 | 3072? | 262K | ✓ (16) | ✓ (12) |

### 量化规格

| 量化类型 | Bits | Group Size | 压缩比 | 精度损失 |
|----------|------|------------|--------|----------|
| INT4 | 4 | 64 | 8x | Minimal |
| BF16 | 16 | - | 2x | None |
| F32 | 32 | - | 1x | None |

---

## 模型转换指南

### 从 HuggingFace 转换

**步骤**:
1. Download original model
2. Quantize to 4-bit (if needed)
3. Convert to safetensors format
4. Organize config files
5. Load with MarkBase-12B

**工具**:
- `safetensors` Python library
- `transformers` (for config)
- Custom quantization scripts

**示例转换脚本**:
```python
from safetensors.torch import save_file
from transformers import AutoModelForCausalLM

# Load original model
model = AutoModelForCausalLM.from_pretrained("model_name")

# Quantize (custom implementation)
quantized_model = quantize_model(model, bits=4, group_size=64)

# Save as safetensors
save_file(quantized_model.state_dict(), "model.safetensors")
```

### 模型文件组织

**目录结构**:
```
model_dir/
├── model.safetensors (or model-00001-of-00002.safetensors)
├── model.safetensors.index.json (if sharded)
├── config.json
├── tokenizer.json
├── tokenizer_config.json
├── generation_config.json
├── processor_config.json (for multimodal)
└── chat_template.jinja (optional)
```

---

## 自定义模型支持

### 添加新模型

**要求**:
1. Gemma-4 architecture family
2. 4-bit quantized weights
3. Safetensors format
4. Compatible config.json

**修改代码**:
```swift
// Sources/G12B/Model.swift
// Adjust architecture parameters if needed

public init(modelDir: String, engine: MarkBaseEngine, maxContextLength: Int) throws {
    // Load custom config
    let config = try loadConfig(modelDir)
    
    // Initialize based on config
    self.numHiddenLayers = config.num_hidden_layers
    self.hiddenSize = config.hidden_size
    ...
}
```

### 模型配置适配

**config.json 适配器**:
```swift
struct ModelConfig: Codable {
    let model_type: String
    let architectures: [String]
    let hidden_size: Int
    let num_hidden_layers: Int
    let vocab_size: Int
    
    // Optional: Vision config
    let vision_config: VisionConfig?
    
    // Optional: Audio config  
    let audio_config: AudioConfig?
}
```

---

## 模型性能对比

### 单设备性能

| 模型 | 推理速度 | 内存占用 | 启动时间 |
|------|----------|----------|----------|
| 4B | ~50 tok/s | ~2GB | ~30s |
| 12B | ~30 tok/s | ~4GB | ~90s |
| E4B-MarkBase | ~25 tok/s | ~6GB | ~90s |

### 分布式性能

| 模型 | Distributed | Bandwidth | Latency |
|------|-------------|-----------|---------|
| 12B | 658 tok/s | 5761 MB/s | Low |

---

## 模型限制

### 当前限制

1. **架构限制**:
   - 仅支持 Gemma-4 family
   - 需要 4-bit quantization
   - Safetensors format only

2. **配置要求**:
   - 必须有完整的 config.json
   - Tokenizer 文件必需
   - Quantization config 需要

3. **Multimodal限制**:
   - Vision Tower 需要 safetensors 中的权重
   - Audio Tower 需要特定架构
   - 测试时 output quality 需验证

### 未来扩展

**计划支持**:
- 其他架构（LLaMA, Mistral, etc）
- 8-bit quantization
- FP16 weights
- 更多 tokenizer 格式

---

## 推荐模型来源

### HuggingFace Models

**Gemma-4 相关**:
- `google/gemma-4-4b-it` (instruction tuned)
- `google/gemma-4-12b-it`
- Custom variants (MarkBase, etc)

**下载方法**:
```bash
# Using huggingface-cli
huggingface-cli download model_name --local-dir ./model

# Using Python
from huggingface_hub import snapshot_download
snapshot_download("model_name", local_dir="./model")
```

### 本地模型

**自定义训练模型**:
- 训练后转换为 safetensors
- 量化到 4-bit
- 组织配置文件
- 加载测试

---

## 使用示例

### 选择并加载模型

**E4B-MarkBase (Multimodal)**:
```bash
swift run G12BServer /models/E4B-MarkBase 8080 markbase

curl -X POST http://localhost:8080/v1/multimodal/chat/completions \
  -d '{"messages":[{"role":"user","content":[{"type":"text","text":"Describe"},{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}]}]}'
```

**Gemma-4-4B-IT (Text-only)**:
```bash
swift run G12BServer /models/gemma-4-4b-it 8080 gemma-4b

curl -X POST http://localhost:8080/v1/chat/completions \
  -d '{"messages":[{"role":"user","content":"Hello"}]}'
```

**Custom Model**:
```bash
swift run G12BServer /models/my-custom-model 8080 custom

# Test if architecture is compatible
swift test --filter testModelLoading
```

---

## 故障排除

### 模型加载失败

**常见错误**:
```
Error: Model not found
→ Check model_dir path is correct

Error: Config not found
→ Ensure config.json exists

Error: Unsupported architecture
→ Check model_type in config.json

Error: Quantization mismatch
→ Verify bits=4, group_size=64
```

### 配置检查

**验证配置**:
```bash
# Check config.json
jq '.' model_dir/config.json

# Verify architecture
jq '.model_type, .architectures' model_dir/config.json

# Check quantization
jq '.quantization_config' model_dir/config.json
```

---

## 总结

**支持的模型类型**:
- ✓ Gemma-4 family (E4B, 12B, MarkBase)
- ✓ 4-bit quantized safetensors
- ✓ Multimodal (Vision + Audio)

**选择建议**:
- 纯文本: Gemma-4-4B-IT
- 视觉理解: E4B-MarkBase
- 音频理解: MarkBase-12B
- 分布式: 12B variants

**加载方法**:
```bash
swift run G12BServer <model_dir> <port> <model_id>
```

**要求**:
- Safetensors weights
- Complete config files
- 4-bit quantization
- Gemma-4 architecture

---

**文档生成**: June 19, 2026
**支持模型**: Gemma-4 Family
**格式要求**: Safetensors + Config