feat: add S3-compatible object storage backend

Add a new backend store that enables iSCSI targets backed by
S3-compatible object storage (AWS S3, MinIO, Ceph RGW, etc.).

The implementation uses a chunked storage strategy where the virtual
block device is divided into fixed-size chunks (default 4 MiB), each
stored as an independent S3 object. This enables efficient random
read/write access on top of object storage.

Key features:
- Chunked storage with configurable chunk size
- Sparse device support (unwritten chunks treated as zeros)
- Concurrent multi-chunk I/O via errgroup
- Per-chunk locking for safe read-modify-write
- AWS SDK v2 with default credential chain
- In-process gofakes3 test server (no Docker needed)
- 12 unit tests + 2 integration tests

Also updates CI workflow to run S3 backend tests and updates
README with S3 backend documentation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Lei Xue
2026-03-16 16:22:57 +08:00
parent 67950263a3
commit 76ab15b0df
13 changed files with 1266 additions and 8 deletions

View File

@@ -46,7 +46,7 @@ func NewSCSILu(bs *config.BackendStorage) (*api.SCSILu, error) {
// Validate backend type, default to file if unknown
switch backendType {
case "file", "iouring", "ceph", "null", "RemBs":
case "file", "iouring", "ceph", "null", "RemBs", "s3":
// Valid types
default:
// Unknown type, treat entire path as file path
@@ -67,6 +67,7 @@ func NewSCSILu(bs *config.BackendStorage) (*api.SCSILu, error) {
Storage: backing,
BlockShift: bs.BlockShift,
UUID: bs.DeviceID,
BackendConfig: bs,
}
err = backing.Open(lu, backendPath)