Initial commit: E4B-MarkBase model integration with passing tests
Some checks failed
CI / build-and-test (push) Has been cancelled
Some checks failed
CI / build-and-test (push) Has been cancelled
- E4B-MarkBase model (42 layers, 4.4GB) loaded successfully - All Phase 1-6 tests passed (model loading, forward pass, vision/audio towers, token generation, performance) - All stress tests passed (5/5 in 127.6s) - Concurrent inference - Memory stress (67.5 tok/s, 0 NaN) - Continuous generation - Batch processing - Long-running stability - Swift Metal inference engine with multimodal support
This commit is contained in:
MarkBase Admin
436
26B测试结果报告.md
Normal file
436
26B测试结果报告.md
Normal file
@@ -0,0 +1,436 @@
|
||||
# Gemma-4 26B 测试结果报告
|
||||
|
||||
## 测试状态: 需要格式适配 ⚠️
|
||||
|
||||
**测试时间**: June 19, 2026
|
||||
**模型位置**: `/Users/accusys/.cache/huggingface/hub/models--mlx-community--gemma-4-26b-a4b-mxfp4/`
|
||||
**模型大小**: 14.8 GB (3 shards)
|
||||
|
||||
---
|
||||
|
||||
## 测试结果
|
||||
|
||||
### 文件检查 ✓
|
||||
```
|
||||
✓ Config.json: 存在
|
||||
✓ Tokenizer.json: 30 MB
|
||||
✓ Weights shard 1: 5063 MB
|
||||
✓ Weights shard 2: 5075 MB
|
||||
✓ Weights shard 3: 4011 MB
|
||||
✓ Total: 1283 tensors
|
||||
```
|
||||
|
||||
### 加载尝试 ⚠️
|
||||
```
|
||||
✓ Engine created
|
||||
✓ Found 3 safetensors shards
|
||||
✗ Error: unsupportedDtype("Embed tokens not quantized")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 问题分析
|
||||
|
||||
### 主要问题
|
||||
|
||||
**错误**: `Embed tokens not quantized`
|
||||
|
||||
**原因**: MLX 格式与我们的格式不兼容
|
||||
|
||||
#### 具体差异
|
||||
|
||||
**1. 权重命名差异**
|
||||
```
|
||||
MLX 格式:
|
||||
language_model.model.embed_tokens.weight
|
||||
language_model.model.layers.0.experts.switch_glu.down_proj.weight
|
||||
language_model.model.layers.0.input_layernorm.weight
|
||||
|
||||
我们的格式:
|
||||
embed_tokens.weight
|
||||
layers.0.down_proj.weight
|
||||
layers.0.input_layernorm.weight
|
||||
```
|
||||
|
||||
**2. Embed tokens 格式**
|
||||
```
|
||||
MLX 26B:
|
||||
embed_tokens.weight: uint32 [262144, 352]
|
||||
embed_tokens.scales: uint8 [262144, 88]
|
||||
|
||||
我们期望:
|
||||
embed_tokens.weight: uint32 (quantized)
|
||||
embed_tokens.scales: uint32 (BF16 scales)
|
||||
embed_tokens.biases: uint32 (BF16 biases)
|
||||
```
|
||||
|
||||
**3. MoE 结构**
|
||||
```
|
||||
MLX 26B 有 MoE (Mixture of Experts):
|
||||
layers.0.experts.switch_glu.down_proj
|
||||
layers.0.experts.switch_glu.gate_proj
|
||||
layers.0.experts.switch_glu.up_proj
|
||||
|
||||
我们的代码不支持 MoE 专家路由
|
||||
```
|
||||
|
||||
**4. Config 结构**
|
||||
```
|
||||
MLX config:
|
||||
{
|
||||
"text_config": {
|
||||
"hidden_size": 2816,
|
||||
"num_hidden_layers": ?,
|
||||
"enable_moe_block": true,
|
||||
...
|
||||
}
|
||||
}
|
||||
|
||||
我们期望:
|
||||
{
|
||||
"hidden_size": 2816,
|
||||
"num_hidden_layers": ?,
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 详细对比
|
||||
|
||||
### 模型架构
|
||||
|
||||
**Gemma-4 26B MLX**:
|
||||
```
|
||||
Model type: gemma4
|
||||
Architecture: Gemma4ForConditionalGeneration
|
||||
Hidden size: 2816 (比 12B 的 2560 大)
|
||||
Intermediate size: 2112
|
||||
MoE blocks: enabled
|
||||
Experts: 128 experts per layer (推测)
|
||||
```
|
||||
|
||||
**我们的 E4B-MarkBase**:
|
||||
```
|
||||
Model type: gemma4
|
||||
Architecture: Gemma4ForConditionalGeneration
|
||||
Hidden size: 2560
|
||||
Intermediate size: 10240
|
||||
MoE: disabled (dense layers)
|
||||
```
|
||||
|
||||
### 权重对比
|
||||
|
||||
| Component | MLX 26B | 我们的 E4B |
|
||||
|-----------|---------|------------|
|
||||
| Embed tokens | uint32 + uint8 scales | uint32 + BF16 scales/biases |
|
||||
| Layers | language_model.model.layers.X | layers.X |
|
||||
| MoE | experts.switch_glu | dense MLP |
|
||||
| Vision | embed_vision.embedding_projection | vision_tower.X |
|
||||
|
||||
### 格式差异
|
||||
|
||||
**量化格式**:
|
||||
```
|
||||
MLX mxfp4:
|
||||
- weight: uint32 (packed 4-bit)
|
||||
- scales: uint8 (8-bit)
|
||||
- 无 biases
|
||||
|
||||
我们的标准 4-bit:
|
||||
- weight: uint32 (packed, group_size=64)
|
||||
- scales: uint32 (BF16)
|
||||
- biases: uint32 (BF16)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 解决方案
|
||||
|
||||
### 方案 1: 转换模型格式 (推荐)
|
||||
|
||||
**步骤**:
|
||||
|
||||
#### 1. 下载并转换
|
||||
```python
|
||||
from safetensors.torch import load_file, save_file
|
||||
import torch
|
||||
|
||||
# Load MLX model
|
||||
mlx_dir = "/Users/accusys/.cache/huggingface/hub/models--mlx-community--gemma-4-26b-a4b-mxfp4"
|
||||
weights = {}
|
||||
for shard in ["model-00001-of-00003.safetensors", ...]:
|
||||
w = load_file(f"{mlx_dir}/{shard}")
|
||||
weights.update(w)
|
||||
|
||||
# Rename weights
|
||||
renamed = {}
|
||||
for key, tensor in weights.items():
|
||||
# Remove language_model.model prefix
|
||||
new_key = key.replace("language_model.model.", "")
|
||||
renamed[new_key] = tensor
|
||||
|
||||
# Convert MoE to dense (可选)
|
||||
# 或保留 MoE 并实现路由
|
||||
|
||||
# Convert scales format
|
||||
# uint8 → BF16 uint32
|
||||
|
||||
# Save as single file
|
||||
save_file(renamed, "gemma-4-26b-converted.safetensors")
|
||||
```
|
||||
|
||||
#### 2. 创建适配的 config.json
|
||||
```json
|
||||
{
|
||||
"model_type": "gemma4",
|
||||
"architectures": ["Gemma4ForConditionalGeneration"],
|
||||
"hidden_size": 2816,
|
||||
"num_hidden_layers": 42,
|
||||
"vocab_size": 262144,
|
||||
"quantization_config": {
|
||||
"bits": 4,
|
||||
"group_size": 64
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. 测试加载
|
||||
```bash
|
||||
swift run G12BServer /path/to/converted-26b 8080 gemma-26b
|
||||
```
|
||||
|
||||
**优点**:
|
||||
- ✓ 可以加载
|
||||
- ✓ 性能优化
|
||||
- ✓ 与现有代码兼容
|
||||
|
||||
**缺点**:
|
||||
- 需要转换时间
|
||||
- MoE 仍需额外实现
|
||||
- 需要足够 memory
|
||||
|
||||
### 方案 2: 适配代码支持 MLX
|
||||
|
||||
**需要修改**:
|
||||
|
||||
#### 1. 权重加载
|
||||
```swift
|
||||
// Sources/G12B/Model.swift
|
||||
|
||||
// 支持两种命名格式
|
||||
let weightName = {
|
||||
if tensorName.hasPrefix("language_model.model.") {
|
||||
return tensorName.replacing("language_model.model.", with: "")
|
||||
}
|
||||
return tensorName
|
||||
}()
|
||||
```
|
||||
|
||||
#### 2. Scales 格式
|
||||
```swift
|
||||
// 支持 uint8 scales
|
||||
if scalesTensor.dtype == .uint8 {
|
||||
// 转换为 BF16
|
||||
scales = convertUint8ToBfloat16(scalesTensor)
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. MoE 支持
|
||||
```swift
|
||||
// 新增 MoE 路由实现
|
||||
struct MoERouter {
|
||||
func route(input: MTLBuffer, experts: [Expert]) -> MTLBuffer {
|
||||
// 专家路由逻辑
|
||||
}
|
||||
}
|
||||
|
||||
struct Expert {
|
||||
let down_proj: QuantizedWeights
|
||||
let gate_proj: QuantizedWeights
|
||||
let up_proj: QuantizedWeights
|
||||
}
|
||||
```
|
||||
|
||||
**优点**:
|
||||
- ✓ 直接支持 MLX
|
||||
- ✓ 无需转换
|
||||
- ✓ 支持更多模型
|
||||
|
||||
**缺点**:
|
||||
- 需要较多代码修改
|
||||
- MoE 实现复杂
|
||||
- 测试工作量
|
||||
|
||||
### 方案 3: 下载标准版本
|
||||
|
||||
**等待官方或社区提供**:
|
||||
- 标准 4-bit quantized 格式
|
||||
- 无 MoE 或 MoE 已转换
|
||||
- 命名符合标准
|
||||
|
||||
**来源**:
|
||||
- HuggingFace 标准量化版本
|
||||
- 自行量化官方模型
|
||||
- 社区转换版本
|
||||
|
||||
**优点**:
|
||||
- ✓ 无需修改代码
|
||||
- ✓ 直接可用
|
||||
- ✓ 官方支持
|
||||
|
||||
**缺点**:
|
||||
- 可能不存在
|
||||
- 需要等待
|
||||
- 需要自己量化
|
||||
|
||||
---
|
||||
|
||||
## Memory 需求估算
|
||||
|
||||
### 26B Memory 分析
|
||||
|
||||
**权重大小**:
|
||||
```
|
||||
26B parameters × 0.5 bytes (4-bit) = 13 GB
|
||||
Embed tokens (可能未量化): +1 GB
|
||||
Vision tower: +0.5 GB
|
||||
Total weights: ~14.5 GB
|
||||
```
|
||||
|
||||
**运行时 Memory**:
|
||||
```
|
||||
Weights: 14.5 GB
|
||||
KV Cache (128 context): 0.5 GB
|
||||
Activations: 1-2 GB
|
||||
Total: ~17 GB
|
||||
```
|
||||
|
||||
**Mac 要求**:
|
||||
```
|
||||
M3 Pro (36GB): ✓ 充足
|
||||
M3 Max (48GB): ✓ 充足
|
||||
M4/M5 (64GB+): ✓ 完全充足
|
||||
M1/M2 Max (24-32GB): ⚠ 勉强
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 推荐路径
|
||||
|
||||
### 立即可行
|
||||
|
||||
**短期 (1-2天)**:
|
||||
- 转换现有 MLX 26B 为标准格式
|
||||
- 转换 scales uint8 → BF16
|
||||
- 重命名权重
|
||||
- 测试加载
|
||||
|
||||
### 长期支持
|
||||
|
||||
**中期 (1-2周)**:
|
||||
- 实现 MLX 格式直接支持
|
||||
- 实现 uint8 scales 支持
|
||||
- 权重命名自动适配
|
||||
|
||||
**长期 (1-2月)**:
|
||||
- 实现完整 MoE 支持
|
||||
- 专家路由优化
|
||||
- 分布式 MoE 推理
|
||||
|
||||
---
|
||||
|
||||
## 下一步行动
|
||||
|
||||
### Option A: 快速转换 (推荐)
|
||||
|
||||
**1. 编写转换脚本** (Python):
|
||||
```bash
|
||||
python convert_mlx_26b.py \
|
||||
--input ~/.cache/huggingface/hub/models--mlx-community--gemma-4-26b-a4b-mxfp4 \
|
||||
--output ~/models/gemma-4-26b-standard \
|
||||
--rename \
|
||||
--convert-scales
|
||||
```
|
||||
|
||||
**2. 测试加载**:
|
||||
```bash
|
||||
swift test --filter test26BModelLoading
|
||||
```
|
||||
|
||||
**3. 性能测试**:
|
||||
```bash
|
||||
swift run G12BServer ~/models/gemma-4-26b-standard 8080 gemma-26b
|
||||
```
|
||||
|
||||
### Option B: 代码适配
|
||||
|
||||
**1. 支持双重命名**:
|
||||
```swift
|
||||
// 修改 Model.swift 支持两种格式
|
||||
```
|
||||
|
||||
**2. uint8 scales 转换**:
|
||||
```swift
|
||||
// 在加载时转换格式
|
||||
```
|
||||
|
||||
**3. 测试验证**:
|
||||
```bash
|
||||
swift test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 结论
|
||||
|
||||
**当前状态**: 26B 模型存在但格式不兼容
|
||||
|
||||
**问题**: MLX 格式 vs 我们的标准格式
|
||||
|
||||
**解决方案**:
|
||||
- ✓ 方案1: 转换格式 (最快)
|
||||
- ⚠️ 方案2: 适配代码 (需要工作量)
|
||||
- ⏳ 方案3: 等待标准版本 (可能不存在)
|
||||
|
||||
**推荐**: **方案 1 - 转换格式**
|
||||
|
||||
**预计时间**: 1-2天完成转换和测试
|
||||
|
||||
**Memory 要求**: M3 Pro/Max 或更高 (36GB+)
|
||||
|
||||
---
|
||||
|
||||
## 附录
|
||||
|
||||
### MLX 权重列表 (部分)
|
||||
|
||||
```
|
||||
language_model.model.embed_tokens.weight [262144, 352] uint32
|
||||
language_model.model.embed_tokens.scales [262144, 88] uint8
|
||||
language_model.model.layers.0.experts.switch_glu.down_proj.weight [128, 2816, 88] uint32
|
||||
language_model.model.layers.0.experts.switch_glu.down_proj.scales [128, 2816, 22] uint8
|
||||
language_model.model.layers.0.input_layernorm.weight [2816] bfloat16
|
||||
language_model.model.layers.0.layer_scalar [1] bfloat16
|
||||
...
|
||||
embed_vision.embedding_projection.weight [...] uint32
|
||||
embed_vision.embedding_projection.scales [...] uint8
|
||||
```
|
||||
|
||||
### 需要的转换脚本功能
|
||||
|
||||
**Python script**:
|
||||
1. Load MLX safetensors shards
|
||||
2. Rename weights (remove language_model.model prefix)
|
||||
3. Convert uint8 scales to BF16
|
||||
4. Flatten MoE structure (可选)
|
||||
5. Merge into single safetensors
|
||||
6. Generate standard config.json
|
||||
7. Copy tokenizer files
|
||||
|
||||
---
|
||||
|
||||
**报告生成**: June 19, 2026
|
||||
**测试结果**: 格式不兼容，需要转换
|
||||
**建议**: 转换 MLX 格式为标准格式
|
||||
|
||||
Reference in New Issue
Block a user