markbase/docs/FUSE_POC_TEST.md

# MarkBase FUSE POC Test Plan

## Overview

**Objective**: Verify FUSE-T installation, basic functionality, and backend selection before full implementation.

**Scope**: Phase 1 POC verification (Day 1-2)

**Environment**: M4 Mac mini, macOS 26.4.1, NVMe storage

---

## Test Environment Setup

### Hardware Configuration

| Component | Specification |
|-----------|--------------|
| **CPU** | Apple M4, 10 physical cores |
| **RAM** | 16 GB unified memory |
| **Storage** | KIOXIA NVMe 2TB (PCIe) |
| **OS** | macOS 26.4.1 (Sequoia) |
| **Network** | Thunderbolt Ethernet |

### Software Dependencies

```bash
# Required tools
brew install fuse-t                # FUSE-T installation
brew install rustup                # Rust toolchain (already installed)

# Optional test tools
brew install ajadatacalc           # AJA Data Calculator (storage estimation)
```

### AJA System Test Installation

```bash
# Download AJA System Test
curl -L -o ~/Downloads/AJA_System_Test.dmg \
  "https://www.aja.com/en/products/aja-system-test"

# Install
hdiutil attach ~/Downloads/AJA_System_Test.dmg
cp -R /Volumes/AJA\ System\ Test/*.app /Applications/
hdiutil detach /Volumes/AJA\ System\ Test

# Verify installation
ls -la /Applications/AJA\ System\ Test.app
```

---

## Test 1: FUSE-T Installation Verification

### Objective

Verify FUSE-T can be installed and basic mount/unmount operations work.

### Steps

```bash
# Step 1: Install FUSE-T
brew install macos-fuse-t/homebrew-cask/fuse-t

# Step 2: Verify installation
ls -la /usr/local/bin/fuse-t
fuse-t --version

# Step 3: Check NFS server
ls -la /usr/local/bin/nfs-t
nfs-t --version

# Step 4: Verify mount capability
mount | grep fuse  # Should show no existing FUSE mounts
```

### Expected Results

| Check | Expected | Actual | Pass/Fail |
|-------|----------|--------|-----------|
| fuse-t binary exists | `/usr/local/bin/fuse-t` | TBD | TBD |
| fuse-t version | >=1.2.6 | TBD | TBD |
| nfs-t binary exists | `/usr/local/bin/nfs-t` | TBD | TBD |
| No existing FUSE mounts | Empty output | TBD | TBD |

### Failure Handling

**If installation fails:**
1. Check System Settings → Privacy & Security → Network Volumes permission
2. Manual download: https://github.com/macos-fuse-t/fuse-t/releases
3. Install from PKG: `sudo installer -pkg fuse-t-macos-installer-1.2.6.pkg -target /`

---

## Test 2: Hello FUSE POC

### Objective

Create minimal FUSE filesystem to verify mount/unmount works.

### Implementation

```rust
// src/fuse/poc_hello.rs
use fuse::{Filesystem, FileAttr, FileType};
use time::Timespec;
use std::path::Path;

pub struct HelloFs;

impl Filesystem for HelloFs {
    fn lookup(&mut self, _parent: u64, name: &str, reply: fuse::ReplyEntry) {
        if name == "hello.txt" {
            let attr = FileAttr {
                ino: 2,
                size: 13,
                blocks: 0,
                atime: Timespec::new(0, 0),
                mtime: Timespec::new(0, 0),
                ctime: Timespec::new(0, 0),
                crtime: Timespec::new(0, 0),
                kind: FileType::RegularFile,
                perm: 0o644,
                nlink: 1,
                uid: 0,
                gid: 0,
                rdev: 0,
                flags: 0,
            };
            reply.entry(&Timespec::new(0, 0), &attr, 0);
        } else {
            reply.error(libc::ENOENT);
        }
    }
    
    fn getattr(&mut self, ino: u64, reply: fuse::ReplyAttr) {
        match ino {
            1 => {
                // Root directory
                let attr = FileAttr {
                    ino: 1,
                    size: 0,
                    blocks: 0,
                    atime: Timespec::new(0, 0),
                    mtime: Timespec::new(0, 0),
                    ctime: Timespec::new(0, 0),
                    crtime: Timespec::new(0, 0),
                    kind: FileType::Directory,
                    perm: 0o755,
                    nlink: 2,
                    uid: 0,
                    gid: 0,
                    rdev: 0,
                    flags: 0,
                };
                reply.attr(&Timespec::new(0, 0), &attr);
            }
            2 => {
                // hello.txt
                let attr = FileAttr {
                    ino: 2,
                    size: 13,
                    blocks: 0,
                    atime: Timespec::new(0, 0),
                    mtime: Timespec::new(0, 0),
                    ctime: Timespec::new(0, 0),
                    crtime: Timespec::new(0, 0),
                    kind: FileType::RegularFile,
                    perm: 0o644,
                    nlink: 1,
                    uid: 0,
                    gid: 0,
                    rdev: 0,
                    flags: 0,
                };
                reply.attr(&Timespec::new(0, 0), &attr);
            }
            _ => reply.error(libc::ENOENT),
        }
    }
    
    fn read(&mut self, ino: u64, _offset: u64, _size: u32, reply: fuse::ReplyData) {
        if ino == 2 {
            reply.data(&b"Hello from MarkBase FUSE!\n"[..]);
        } else {
            reply.error(libc::ENOENT);
        }
    }
    
    fn readdir(&mut self, ino: u64, _offset: u64, reply: fuse::ReplyDirectory) {
        if ino == 1 {
            reply.add(1, 0, FileType::Directory, ".");
            reply.add(1, 1, FileType::Directory, "..");
            reply.add(2, 2, FileType::RegularFile, "hello.txt");
            reply.ok();
        } else {
            reply.error(libc::ENOENT);
        }
    }
}

pub fn mount_hello_fs(path: &Path) -> Result<(), Box<dyn std::error::Error>> {
    let fs = HelloFs;
    fuse::mount(fs, path, &[])?;
    Ok(())
}
```

### Test Commands

```bash
# Create test directory
mkdir -p /tmp/fuse_test

# Mount Hello FUSE
cargo run -- fuse --poc --mount --dir /tmp/fuse_test

# Verify mount
mount | grep fuse_test
ls -la /tmp/fuse_test
cat /tmp/fuse_test/hello.txt

# Expected output: "Hello from MarkBase FUSE!\n"

# Unmount
cargo run -- fuse --poc --unmount --dir /tmp/fuse_test
umount /tmp/fuse_test  # Manual fallback
```

### Expected Results

| Check | Expected | Actual | Pass/Fail |
|-------|----------|--------|-----------|
| Mount succeeds | Exit code 0 | TBD | TBD |
| Mount visible in `mount` | `fuse_test` entry | TBD | TBD |
| Directory listing | `hello.txt` visible | TBD | TBD |
| File read | "Hello from MarkBase FUSE!\n" | TBD | TBD |
| Unmount succeeds | Exit code 0 | TBD | TBD |
| Mount removed | No `fuse_test` in `mount` | TBD | TBD |

---

## Test 3: Backend Selection Verification

### Objective

Test both NFSv4 and FSKit backends to verify performance difference.

### Backend Detection

```rust
// src/fuse/backend.rs
use std::env;

pub fn detect_macos_version() -> String {
    env::var("MACOS_VERSION").unwrap_or_else(|_| {
        // Fallback: parse from system
        let output = Command::new("sw_vers")
            .arg("-productVersion")
            .output()
            .expect("Failed to get macOS version");
        String::from_utf8_lossy(&output.stdout).trim().to_string()
    })
}

pub fn select_backend() -> BackendType {
    let version = detect_macos_version();
    
    if version.starts_with("26") {
        BackendType::Fskit  // macOS 26+ supports FSKit
    } else {
        BackendType::Nfs4   // Older macOS uses NFSv4
    }
}
```

### Test NFSv4 Backend

```bash
# Force NFSv4 backend
cargo run -- fuse --poc --mount --dir /tmp/fuse_nfs --backend nfs

# Performance test
time ls -la /tmp/fuse_nfs
time cat /tmp/fuse_nfs/hello.txt

# Unmount
cargo run -- fuse --poc --unmount --dir /tmp/fuse_nfs
```

### Test FSKit Backend

```bash
# Force FSKit backend (macOS 26+)
cargo run -- fuse --poc --mount --dir /tmp/fuse_fskit --backend fskit

# Performance test
time ls -la /tmp/fuse_fskit
time cat /tmp/fuse_fskit/hello.txt

# Unmount
cargo run -- fuse --poc --unmount --dir /tmp/fuse_fskit
```

### Performance Comparison

| Backend | ls latency | cat latency | Notes |
|---------|------------|--------------|-------|
| NFSv4 | TBD | TBD | Baseline |
| FSKit | TBD | TBD | Target: 40% faster |
| Direct disk | TBD | TBD | Reference |

---

## Test 4: Basic Read/Write Operations

### Objective

Verify basic file operations work through FUSE mount.

### Test Script

```bash
#!/bin/bash
# tests/fuse_basic_ops.sh

MOUNT_DIR="/tmp/fuse_ops_test"

# Setup
mkdir -p "$MOUNT_DIR"
cargo run -- fuse --poc --mount --dir "$MOUNT_DIR"

# Test 1: Directory listing
echo "=== Test 1: Directory Listing ==="
ls -la "$MOUNT_DIR"

# Test 2: File read
echo "=== Test 2: File Read ==="
cat "$MOUNT_DIR/hello.txt"

# Test 3: File stat
echo "=== Test 3: File Stat ==="
stat "$MOUNT_DIR/hello.txt"

# Cleanup
cargo run -- fuse --poc --unmount --dir "$MOUNT_DIR"
rm -rf "$MOUNT_DIR"

echo "All tests completed"
```

### Expected Results

| Operation | Expected Output | Pass/Fail |
|-----------|-----------------|-----------|
| Directory listing | Shows `hello.txt` | TBD |
| File read | "Hello from MarkBase FUSE!\n" | TBD |
| File stat | ino=2, size=13, perm=644 | TBD |
| Unmount | Exit code 0 | TBD |

---

## Test 5: AJA System Test Baseline

### Objective

Measure raw disk performance before FUSE implementation.

### AJA Test Configuration

| Setting | Value |
|---------|-------|
| **Target** | `/System/Volumes/Data` (NVMe) |
| **Resolution** | 4K (3840×2160) |
| **Codec** | ProRes 4444 |
| **Frame Rate** | 60 fps |
| **Duration** | 10 seconds |
| **Mode** | Write test |

### Test Procedure

```bash
# Open AJA System Test
open -a "AJA System Test"

# Manual steps:
1. Select target: /System/Volumes/Data
2. Configure: 4K ProRes 4444, 60fps
3. Run Write test (10 seconds)
4. Record results:
   - Write speed: ___ MB/s
   - Frame rate achieved: ___ fps
   - Disk usage: ___ %
```

### dd Baseline Test

```bash
# Write test (6GB)
time dd if=/dev/zero of=/System/Volumes/Data/test_baseline.bin bs=1M count=6000

# Expected: ~1.5GB/s (NVMe raw speed)
# Record: ___ MB/s

# Cleanup
rm /System/Volumes/Data/test_baseline.bin
```

### Expected Results

| Test | Expected Speed | Actual Speed | Pass/Fail |
|------|----------------|--------------|-----------|
| AJA 4K ProRes 4444 Write | >=1500 MB/s | TBD | TBD |
| dd write (6GB) | >=1500 MB/s | TBD | TBD |
| AJA frame rate | 60 fps sustained | TBD | TBD |

---

## Test 6: Rust FUSE Library Verification

### Objective

Verify Rust FUSE library compatibility with FUSE-T.

### Dependency Check

```toml
# Cargo.toml
[dependencies]
fuse = "0.3"  # Or fuse-t specific binding
time = "0.3"
libc = "0.2"
```

### Compilation Test

```bash
# Add fuse dependency
cargo add fuse

# Compile check
cargo check

# Expected: No compilation errors
```

### Mount Options Verification

```rust
// src/fuse/options.rs
use fuse::MountOption;

pub fn get_mount_options(backend: BackendType) -> Vec<MountOption> {
    match backend {
        BackendType::Nfs4 => vec![
            MountOption::FSName("markbase"),
            MountOption::Noatime,
        ],
        BackendType::Fskit => vec![
            MountOption::FSName("markbase"),
            MountOption::Noatime,
        ],
    }
}
```

### Expected Results

| Check | Expected | Actual | Pass/Fail |
|-------|----------|--------|-----------|
| Cargo build succeeds | Exit code 0 | TBD | TBD |
| fuse library linked | Binary includes fuse symbols | TBD | TBD |
| Mount options valid | No runtime errors | TBD | TBD |

---

## Test Execution Schedule

### Day 1: Environment Setup

| Time | Task | Duration |
|------|------|----------|
| 09:00 | Install FUSE-T | 15 min |
| 09:15 | Verify installation | 10 min |
| 09:25 | Download AJA System Test | 20 min |
| 09:45 | Install AJA System Test | 10 min |
| 09:55 | Run AJA baseline test | 30 min |
| 10:25 | Document results | 15 min |
| **Total** | | **1.5 hours** |

### Day 1: Hello FUSE POC

| Time | Task | Duration |
|------|------|----------|
| 10:40 | Create src/fuse/poc_hello.rs | 30 min |
| 11:10 | Add fuse dependency | 10 min |
| 11:20 | Compile and test | 20 min |
| 11:40 | Document results | 15 min |
| **Total** | | **1.25 hours** |

### Day 2: Backend Verification

| Time | Task | Duration |
|------|------|----------|
| 09:00 | Test NFSv4 backend | 30 min |
| 09:30 | Test FSKit backend | 30 min |
| 10:00 | Compare performance | 20 min |
| 10:20 | Document findings | 20 min |
| **Total** | | **1.7 hours** |

---

## Test Reporting Template

### FUSE-T Installation Report

```
## Test 1: FUSE-T Installation

**Date**: 2026-05-17
**Environment**: M4 Mac mini, macOS 26.4.1

### Results

| Check | Expected | Actual | Pass/Fail |
|-------|----------|--------|-----------|
| fuse-t binary | Exists | [Record] | [Pass/Fail] |
| fuse-t version | >=1.2.6 | [Record] | [Pass/Fail] |
| nfs-t binary | Exists | [Record] | [Pass/Fail] |

### Issues Found
- [Document any issues]

### Resolution
- [Document solutions applied]
```

### Hello FUSE POC Report

```
## Test 2: Hello FUSE POC

**Date**: 2026-05-17
**Mount Path**: /tmp/fuse_test

### Results

| Check | Expected | Actual | Pass/Fail |
|-------|----------|--------|-----------|
| Mount succeeds | Exit 0 | [Record] | [Pass/Fail] |
| Directory listing | hello.txt | [Record] | [Pass/Fail] |
| File read | "Hello..." | [Record] | [Pass/Fail] |

### Latency Measurements
- Mount time: [Record] ms
- ls time: [Record] ms
- cat time: [Record] ms

### Backend Used
- Backend: [NFSv4/FSKit/Auto]
- Reason: [Auto-detect/Manual]
```

### Performance Baseline Report

```
## Test 5: AJA System Test Baseline

**Date**: 2026-05-17
**Target**: /System/Volumes/Data (NVMe)

### AJA Results

| Codec | Resolution | Frame Rate | Write Speed | Pass/Fail |
|-------|------------|------------|-------------|-----------|
| ProRes 4444 | 4K | 60fps | [Record] MB/s | [Pass/Fail] |

### dd Results

| Test | Block Size | Count | Speed | Pass/Fail |
|------|------------|-------|-------|-----------|
| Write | 1MB | 6000 | [Record] MB/s | [Pass/Fail] |

### Analysis
- NVMe raw speed: [Record] MB/s
- AJA speed: [Record] MB/s
- Overhead: [Calculate] %
```

---

## Failure Recovery Procedures

### FUSE-T Installation Failure

**Symptoms:**
- brew install fails
- Permission denied
- Network volumes not accessible

**Recovery:**
```bash
# Step 1: Check permissions
System Settings → Privacy & Security → Files and Folders → Network Volumes → Enable

# Step 2: Manual install
curl -L -o ~/Downloads/fuse-t.pkg \
  https://github.com/macos-fuse-t/fuse-t/releases/download/1.2.6/fuse-t-macos-installer-1.2.6.pkg
sudo installer -pkg ~/Downloads/fuse-t.pkg -target /

# Step 3: Restart terminal
exec zsh
```

### Mount Failure

**Symptoms:**
- mount command hangs
- Operation not permitted
- Device not found

**Recovery:**
```bash
# Check mount status
mount | grep fuse

# Force unmount
umount -f /tmp/fuse_test

# Check process
ps aux | grep fuse

# Kill if needed
kill -9 [PID]
```

### Compilation Failure

**Symptoms:**
- Cargo build fails
- fuse library not found
- Linking errors

**Recovery:**
```bash
# Check Rust version
rustc --version  # Need >=1.70

# Update toolchain
rustup update stable

# Rebuild dependencies
cargo clean
cargo build
```

---

## Success Criteria

### Phase 1 POC Complete

| Criterion | Threshold | Measurement |
|-----------|-----------|-------------|
| FUSE-T installed | Binary exists | `/usr/local/bin/fuse-t` present |
| Hello FUSE works | Read/write succeeds | `cat /tmp/fuse_test/hello.txt` outputs correctly |
| Backend detected | Auto-select works | macOS 26 → FSKit, older → NFSv4 |
| AJA baseline | >=1500 MB/s | Raw disk write speed |
| Compilation succeeds | Exit code 0 | `cargo build` completes |

### Proceed to Phase 2

**Decision Criteria:**
- All Test 1-6 pass
- No critical failures
- Performance baseline documented
- Backend selection confirmed

---

## Next Steps After POC

### Phase 2: SQLite-backed FUSE (Day 3-5)

1. Implement `MarkBaseFs` struct
2. Integrate file_nodes/file_locations queries
3. Test warren user (12,659 nodes)
4. Implement LRU caching
5. Performance measurement

### Phase 3: Multi-user Concurrent (Day 6-8)

1. Implement `MountManager`
2. Test 10 user parallel mount
3. AJA concurrent write test
4. Stability test (24h)

### Phase 4: Performance Optimization (Day 9-12)

1. Write buffering (64KB chunks)
2. FSKit backend optimization
3. AJA 600MB/s target validation
4. Final documentation

---

**Last Updated**: 2026-05-17
**Version**: 1.0
**Status**: Ready for Execution