markbaseengine/支持模型列表.md

MarkBase-12B 支持的模型列表

模型架构支持

当前支持的模型类型

Gemma-4 系列:

• ✓ Gemma-4 E4B (Early Access 4B)
• ✓ Gemma-4 12B
• ✓ E4B-MarkBase (multimodal variant)
• ✓ MarkBase-12B (multimodal variant)

模型架构:

• Gemma4ForConditionalGeneration (multimodal)
• 42层 Transformer
• 262,144 vocabulary size
• Vision Tower (16 layers)
• Audio Tower (12 layers)

---

模型加载方式

1. 从本地目录加载

基本用法:

swift run G12BServer <model_dir> <port> <model_id>

示例:

# 加载 E4B-MarkBase
swift run G12BServer /Users/accusys/MarkBase12B/models/E4B-MarkBase 8080 markbase-12b

# 加载其他模型
swift run G12BServer /path/to/your/model 8080 custom-model

所需文件:

model_dir/
  model.safetensors - 模型权重（4-bit quantized）
  model.safetensors.index.json - 权重索引（如果分片）
  config.json - 模型配置
  tokenizer.json - Tokenizer
  tokenizer_config.json - Tokenizer配置
  generation_config.json - 生成配置

2. Safetensors 格式要求

权重格式:

• Safetensors binary format
• 4-bit quantization (uint32 packed)
• Group size: 64
• BF16 scales/biases

支持的数据类型:

• INT4 (4-bit quantized)
• BF16 (scales/biases)
• F32 (activations)

3. 配置文件格式

config.json 示例:

{
  "model_type": "gemma4",
  "architectures": ["Gemma4ForConditionalGeneration"],
  "hidden_size": 2560,
  "num_hidden_layers": 42,
  "vocab_size": 262144,
  "num_attention_heads": 8,
  "num_key_value_heads": 2,
  "head_dim": 256,
  "intermediate_size": 10240,
  "max_position_embeddings": 131072,
  "quantization_config": {
    "bits": 4,
    "group_size": 64
  }
}

Vision 配置:

{
  "vision_config": {
    "hidden_size": 768,
    "num_hidden_layers": 16,
    "patch_size": 16,
    "image_size": 224
  }
}

Audio 配置:

{
  "audio_config": {
    "hidden_size": 640,
    "num_hidden_layers": 12,
    "num_mel_bins": 128
  }
}

---

模型选择指南

根据 Use Case 选择

1. 纯文本推理

推荐模型: Gemma-4-4B-IT (Instruction Tuned)

• 适用: 文本生成、对话、问答
• 不需要: Vision/Audio components
• 性能: 更快（无 multimodal overhead）

2. 视觉理解

推荐模型: E4B-MarkBase, MarkBase-12B

• 适用: 图像描述、视觉问答、场景理解
• 需要: Vision Tower (16 layers)
• 输入: 224x224 RGB images

3. 音频理解

推荐模型: MarkBase-12B (with audio tower)

• 适用: 音频描述、语音识别、音频问答
• 需要: Audio Tower (12 layers)
• 输入: Mel spectrograms (128 bands)

4. 多模态推理

推荐模型: E4B-MarkBase, MarkBase-12B

• 适用: Vision + Audio + Text
• 需要: Vision + Audio Towers
• 输入: Images + Audio + Text

5. 分布式推理

推荐模型: 12B variants

• 适用: 跨设备推理、高性能
• 需要: RDMA setup (Thunderbolt 5)
• 性能: 658 tok/s (distributed)

---

模型规格对比

Gemma-4 模型系列

模型	参数量	Layers	Hidden Size	Vocab	Vision	Audio
Gemma-4-4B	4B	42	2560	262K	-	-
Gemma-4-12B	12B	42	3072	262K	-	-
E4B-MarkBase	~12B	42	2560	262K	✓ (16)	✓ (12)
MarkBase-12B	~12B	42	3072?	262K	✓ (16)	✓ (12)

量化规格

量化类型	Bits	Group Size	压缩比	精度损失
INT4	4	64	8x	Minimal
BF16	16	-	2x	None
F32	32	-	1x	None

---

模型转换指南

从 HuggingFace 转换

步骤:

1. Download original model
2. Quantize to 4-bit (if needed)
3. Convert to safetensors format
4. Organize config files
5. Load with MarkBase-12B

工具:

• safetensors Python library
• transformers (for config)
• Custom quantization scripts

示例转换脚本:

from safetensors.torch import save_file
from transformers import AutoModelForCausalLM

# Load original model
model = AutoModelForCausalLM.from_pretrained("model_name")

# Quantize (custom implementation)
quantized_model = quantize_model(model, bits=4, group_size=64)

# Save as safetensors
save_file(quantized_model.state_dict(), "model.safetensors")

模型文件组织

目录结构:

model_dir/
├── model.safetensors (or model-00001-of-00002.safetensors)
├── model.safetensors.index.json (if sharded)
├── config.json
├── tokenizer.json
├── tokenizer_config.json
├── generation_config.json
├── processor_config.json (for multimodal)
└── chat_template.jinja (optional)

---

自定义模型支持

添加新模型

要求:

1. Gemma-4 architecture family
2. 4-bit quantized weights
3. Safetensors format
4. Compatible config.json

修改代码:

// Sources/G12B/Model.swift
// Adjust architecture parameters if needed

public init(modelDir: String, engine: MarkBaseEngine, maxContextLength: Int) throws {
    // Load custom config
    let config = try loadConfig(modelDir)
    
    // Initialize based on config
    self.numHiddenLayers = config.num_hidden_layers
    self.hiddenSize = config.hidden_size
    ...
}

模型配置适配

config.json 适配器:

struct ModelConfig: Codable {
    let model_type: String
    let architectures: [String]
    let hidden_size: Int
    let num_hidden_layers: Int
    let vocab_size: Int
    
    // Optional: Vision config
    let vision_config: VisionConfig?
    
    // Optional: Audio config  
    let audio_config: AudioConfig?
}

---

模型性能对比

单设备性能

模型	推理速度	内存占用	启动时间
4B	~50 tok/s	~2GB	~30s
12B	~30 tok/s	~4GB	~90s
E4B-MarkBase	~25 tok/s	~6GB	~90s

分布式性能

模型	Distributed	Bandwidth	Latency
12B	658 tok/s	5761 MB/s	Low

---

模型限制

当前限制

1. 架构限制:

   • 仅支持 Gemma-4 family
   • 需要 4-bit quantization
   • Safetensors format only

2. 配置要求:

   • 必须有完整的 config.json
   • Tokenizer 文件必需
   • Quantization config 需要

3. Multimodal限制:

   • Vision Tower 需要 safetensors 中的权重
   • Audio Tower 需要特定架构
   • 测试时 output quality 需验证

未来扩展

计划支持:

• 其他架构（LLaMA, Mistral, etc）
• 8-bit quantization
• FP16 weights
• 更多 tokenizer 格式

---

推荐模型来源

HuggingFace Models

Gemma-4 相关:

• google/gemma-4-4b-it (instruction tuned)
• google/gemma-4-12b-it
• Custom variants (MarkBase, etc)

下载方法:

# Using huggingface-cli
huggingface-cli download model_name --local-dir ./model

# Using Python
from huggingface_hub import snapshot_download
snapshot_download("model_name", local_dir="./model")

本地模型

自定义训练模型:

• 训练后转换为 safetensors
• 量化到 4-bit
• 组织配置文件
• 加载测试

---

使用示例

选择并加载模型

E4B-MarkBase (Multimodal):

swift run G12BServer /models/E4B-MarkBase 8080 markbase

curl -X POST http://localhost:8080/v1/multimodal/chat/completions \
  -d '{"messages":[{"role":"user","content":[{"type":"text","text":"Describe"},{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}]}]}'

Gemma-4-4B-IT (Text-only):

swift run G12BServer /models/gemma-4-4b-it 8080 gemma-4b

curl -X POST http://localhost:8080/v1/chat/completions \
  -d '{"messages":[{"role":"user","content":"Hello"}]}'

Custom Model:

swift run G12BServer /models/my-custom-model 8080 custom

# Test if architecture is compatible
swift test --filter testModelLoading

---

故障排除

模型加载失败

常见错误:

Error: Model not found
→ Check model_dir path is correct

Error: Config not found
→ Ensure config.json exists

Error: Unsupported architecture
→ Check model_type in config.json

Error: Quantization mismatch
→ Verify bits=4, group_size=64

配置检查

验证配置:

# Check config.json
jq '.' model_dir/config.json

# Verify architecture
jq '.model_type, .architectures' model_dir/config.json

# Check quantization
jq '.quantization_config' model_dir/config.json

---

总结

支持的模型类型:

• ✓ Gemma-4 family (E4B, 12B, MarkBase)
• ✓ 4-bit quantized safetensors
• ✓ Multimodal (Vision + Audio)

选择建议:

• 纯文本: Gemma-4-4B-IT
• 视觉理解: E4B-MarkBase
• 音频理解: MarkBase-12B
• 分布式: 12B variants

加载方法:

swift run G12BServer <model_dir> <port> <model_id>

要求:

• Safetensors weights
• Complete config files
• 4-bit quantization
• Gemma-4 architecture

---

文档生成: June 19, 2026 支持模型: Gemma-4 Family 格式要求: Safetensors + Config