markbase/docs/RAID_DEPLOYMENT_CHECKLIST.md

RAID Production Deployment Checklist

项目： MarkBase RAID 0 Production Deployment 目标： 28 GB/s read bandwidth (4 NVMe disks) 状态： Ready for Deployment

Pre-Deployment Checklist

Hardware Requirements

• Linux Server Available

  • OS: Ubuntu 22.04 LTS or Rocky Linux 9
  • CPU: 8+ cores (AMD EPYC or Intel Xeon)
  • RAM: 64GB DDR5 ECC
  • NVMe slots: 4 PCIe 4.0 lanes per disk

• NVMe SSDs

  • Count: 4 disks minimum
  • Model: Samsung 980 Pro 2TB or WD Black SN850X 2TB
  • Speed: 7000 MB/s read, 5000 MB/s write
  • Health: <10% wear (check nvme smart-log)

• Network (Optional for iSCSI)

  • NIC: 10GbE or 25GbE Mellanox
  • Switch: 10GbE+ capable
  • Cables: SFP+ or RJ45

• Budget Approved

  • Total: $2516 (4-disk config)
  • NVMe SSDs: $716
  • Server: $1500
  • NIC: $200
  • Misc: $100

Software Prerequisites

• Operating System

  # Check OS version
  cat /etc/os-release
  
  # Required: Ubuntu 22.04+ or Rocky Linux 9+
  

• Dependencies Installed

  sudo apt update
  sudo apt install -y mdadm fio sysstat nvme-cli tgt
  

• MarkBase Scripts Available

  • scripts/deploy_raid0_linux.sh ✅
  • scripts/raid0_performance_test.fio ✅
  • scripts/run_raid0_tests.sh ✅
  • scripts/markbase_raid0_integration.sh ✅

• Storage Backup Plan

  • RAID 0 has NO redundancy
  • Backup target identified (RAID 10 array)
  • rsync script prepared

Deployment Steps

Step 1: Hardware Verification (15 min)

# Check NVMe devices
nvme list

# Expected output:
# Node          SN                   Model                    Namespace Usage                      Format       WWN
# /dev/nvme0n1  S5G2NF0N300001P      Samsung SSD 980 PRO 2TB  1         2.00  TB /   2.00  TB    512   B +  0 B   eui.0000000000000002
# /dev/nvme1n1  S5G2NF0N300002P      Samsung SSD 980 PRO 2TB  1         2.00  TB /   2.00  TB    512   B +  0 B   eui.0000000000000003
# /dev/nvme2n1  S5G2NF0N300003P      Samsung SSD 980 PRO 2TB  1         2.00  TB /   2.00  TB    512   B +  0 B   eui.0000000000000004
# /dev/nvme3n1  S5G2NF0N300004P      Samsung SSD 980 PRO 2TB  1         2.00  TB /   2.00  TB    512   B +  0 B   eui.0000000000000005

# Check disk health
for disk in /dev/nvme*n1; do
    echo "=== $disk ==="
    nvme smart-log $disk | grep -E "(temperature|available_spare|percentage_used)"
done

# Expected: percentage_used < 10%

• NVMe list shows 4+ devices
• All disks show <10% wear
• Temperature <70°C

Step 2: RAID 0 Creation (10 min)

# Execute deployment script
sudo bash scripts/deploy_raid0_linux.sh

# Script will:
# 1. Wipe disk metadata
# 2. Create RAID 0 array (/dev/md0)
# 3. Create XFS filesystem
# 4. Mount at /mnt/raid0_media
# 5. Save configuration

• RAID array created (/dev/md0)
• XFS filesystem mounted
• Configuration saved (/etc/mdadm/mdadm.conf)
• Auto-start on reboot enabled

Step 3: Initial Performance Test (5 min)

# Quick dd test
dd if=/dev/zero of=/mnt/raid0_media/test_1g.dat bs=1M count=10240 conv=fdatasync oflag=direct

# Expected: >20 GB/s write speed
# If <15 GB/s: Check PCIe bandwidth, NUMA issues

dd if=/mnt/raid0_media/test_1g.dat of=/dev/null bs=1M count=10240 iflag=direct

# Expected: >28 GB/s read speed

• Write speed >20 GB/s
• Read speed >28 GB/s
• No I/O errors

Step 4: Comprehensive Performance Testing (30 min)

# Run full test suite
sudo bash scripts/run_raid0_tests.sh

# Tests:
# 1. Sequential read/write (1MB blocks)
# 2. 4K video streaming (64KB blocks)
# 3. AJA ProRes equivalent (256KB blocks)
# 4. Concurrent streams (4 streams)
# 5. IOPS saturation (4KB random)

• Sequential read: 28 GB/s ✅
• Sequential write: 20 GB/s ✅
• 4K ProRes: 3200 MB/s (4 streams) ✅
• IOPS: 2400K ✅

Step 5: MarkBase Integration (Optional, 20 min)

# Option A: Direct mount (recommended)
# No additional setup needed
# MarkBase WebDAV uses /mnt/raid0_media directly

# Option B: iSCSI export
sudo bash scripts/markbase_raid0_integration.sh

# Configure iSCSI target:
# Target: iqn.2026-05.com.markbase:raid0.media
# Port: 3260
# Backend: /dev/md0

• WebDAV backend configured
• SQLite integration tested
• File tree sync working

Post-Deployment Verification

Performance Validation

# Monitor real-time I/O
iostat -x /dev/md0 1 5

# Expected metrics:
# tps (transfers per second): >1000
# MB_read/s: >28000
# MB_wrtn/s: >20000
# await (await time): <1ms

• iostat shows expected throughput
• Latency <1ms
• No queue buildup

Stability Test

# 24-hour stress test
fio --name=stress_test \
    --filename=/mnt/raid0_media/stress.dat \
    --size=500G \
    --bs=1M \
    --rw=randrw \
    --direct=1 \
    --numjobs=8 \
    --iodepth=64 \
    --time_based \
    --runtime=86400 \
    --group_reporting

# Expected: No crashes, consistent performance

• 24-hour test passed
• No disk failures
• Performance stable

Integration Test

# Test MarkBase WebDAV
cargo run --bin markbase --release -- display

# Access via browser:
# http://localhost:11438/webdav/warren/

# Test file operations:
# - List directory (PROPFIND)
# - Read file (GET)
# - Write file (PUT)

• WebDAV accessible
• File operations working
• Performance acceptable

Maintenance Schedule

Daily Monitoring

# Check RAID status
mdadm --detail /dev/md0

# Expected: State: clean, Active Devices: 4

# Check disk health
for disk in /dev/nvme*n1; do
    nvme smart-log $disk | grep percentage_used
done

# Expected: <0.01% increase per day

Weekly Backup

# Backup critical data to RAID 10 array
rsync -avh --progress /mnt/raid0_media/critical/ /mnt/raid10_backup/

# Or backup to external storage
rsync -avh --progress /mnt/raid0_media/ /mnt/external_backup/

Monthly Health Check

# Full disk health assessment
nvme smart-log /dev/nvme0n1 > /var/log/nvme_health.log

# Check for:
# - percentage_used (should be <20% after 1 year)
# - media_errors (should be 0)
# - num_err_log_entries (should be low)

Troubleshooting Guide

Common Issues

Issue	Symptom	Solution
Low performance	<15 GB/s read	Check PCIe lanes, NUMA config
Disk failure	mdadm: Faulty	Replace disk immediately (RAID 0 data loss!)
Mount failure	XFS error	Check filesystem: xfs_repair /dev/md0
iSCSI disconnect	Network timeout	Check NIC bonding, switch config

Emergency Procedures

Disk Failure (RAID 0 Critical!)

# RAID 0 has NO redundancy
# Single disk failure = TOTAL DATA LOSS

# Immediate actions:
# 1. Stop all I/O immediately
# 2. Check if data recoverable from backup
# 3. Replace failed disk
# 4. Rebuild RAID 0 from backup
# 5. Verify data integrity

# Prevention: Backup daily to RAID 10 array!

Performance Degradation

# Check bottlenecks
top          # CPU usage
iostat -x 1  # I/O queue
numactl -H   # NUMA topology

# Solutions:
# - Increase iodepth (fio)
# - Use NUMA-aware allocation
# - Check PCIe bandwidth (lspci)

Success Criteria

Performance Targets

• Read bandwidth: ≥28 GB/s (4 × 7000 MB/s)
• Write bandwidth: ≥20 GB/s (4 × 5000 MB/s)
• IOPS: ≥2400K (4 × 600K)
• Latency: <1ms average

Integration Targets

• MarkBase WebDAV working
• SQLite backend functional
• File tree sync operational
• iSCSI export available (optional)

Stability Targets

• 24-hour stress test passed
• No crashes or errors
• Consistent performance
• Daily health check automated

Cost Summary

Item	Cost	Notes
NVMe SSDs (4×2TB)	$716	Samsung 980 Pro or WD Black SN850X
Linux Server	$1500	8-core, 64GB RAM, 4+ NVMe slots
10GbE NIC	$200	Mellanox ConnectX-5
Miscellaneous	$100	Cables, rails, etc.
Total	$2516	4-disk RAID 0 configuration

Performance/Cost Ratio:

• 28 GB/s read / 2516 = 11.2 MB/s/
• 20 GB/s write / 2516 = 8.0 MB/s/

Next Steps After Deployment

1. Production Integration

   • Configure MarkBase WebDAV backend
   • Test file tree synchronization
   • Verify performance in real workload

2. Monitoring Setup

   • Implement daily health check script
   • Configure alerting (disk failure critical!)
   • Setup backup automation

3. Scaling Options

   • Add more NVMe disks (up to 16)
   • Consider RAID 10 for redundancy
   • Implement distributed RAID (future)

4. Documentation Update

   • Record actual performance results
   • Update RAID_INTEGRATION_STATUS.md
   • Create maintenance procedures

---

Prepared by: MarkBase Team Date: 2026-05-19 Version: 1.0 Status: Ready for Production Deployment